04-BERT_Classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "BERT_Classification.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EKOTlwcmxmej",
        "colab_type": "text"
      },
      "source": [
        "# BERT Approach\n",
        "\n",
        "Courtesy Chris McCormick and Nick Ryan"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qWVVHS0OH24-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DEfSbAA4QHas",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "fa8f5a57-3713-4d1d-f766-47e164c729e1"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# Get the GPU device name.\n",
        "device_name = tf.test.gpu_device_name()\n",
        "\n",
        "# The device name should look like the following:\n",
        "if device_name == '/device:GPU:0':\n",
        "    print('Found GPU at: {}'.format(device_name))\n",
        "else:\n",
        "    raise SystemError('GPU device not found')\n",
        "\n",
        "#==========================\n",
        "import torch\n",
        "\n",
        "# If there's a GPU available...\n",
        "if torch.cuda.is_available():    \n",
        "\n",
        "    # Tell PyTorch to use the GPU.    \n",
        "    device = torch.device(\"cuda\")\n",
        "\n",
        "    print('There are %d GPU(s) available.' % torch.cuda.device_count())\n",
        "\n",
        "    print('We will use the GPU:', torch.cuda.get_device_name(0))\n",
        "\n",
        "# If not...\n",
        "else:\n",
        "    print('No GPU available, using the CPU instead.')\n",
        "    device = torch.device(\"cpu\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found GPU at: /device:GPU:0\n",
            "There are 1 GPU(s) available.\n",
            "We will use the GPU: Tesla K80\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqG7FzRVFEIv",
        "colab_type": "text"
      },
      "source": [
        "In order for torch to use the GPU, we need to identify and specify the GPU as the device. Later, in our training loop, we will load data onto the device. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oYsV4H8fCpZ-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "411767f6-cfa2-4b8f-f80b-c153cb0a0385"
      },
      "source": [
        "import torch\n",
        "\n",
        "# If there's a GPU available...\n",
        "if torch.cuda.is_available():    \n",
        "\n",
        "    # Tell PyTorch to use the GPU.    \n",
        "    device = torch.device(\"cuda\")\n",
        "\n",
        "    print('There are %d GPU(s) available.' % torch.cuda.device_count())\n",
        "\n",
        "    print('We will use the GPU:', torch.cuda.get_device_name(0))\n",
        "\n",
        "# If not...\n",
        "else:\n",
        "    print('No GPU available, using the CPU instead.')\n",
        "    device = torch.device(\"cpu\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "There are 1 GPU(s) available.\n",
            "We will use the GPU: Tesla K80\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ElsnSNUridI",
        "colab_type": "text"
      },
      "source": [
        "## 1.2. Installing the Hugging Face Library\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0NmMdkZO8R6q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lxddqmruamSj",
        "colab_type": "text"
      },
      "source": [
        "The code in this notebook is actually a simplified version of the [run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/run_glue.py) example script from huggingface.\n",
        "\n",
        "`run_glue.py` is a helpful utility which allows you to pick which GLUE benchmark task you want to run on, and which pre-trained model you want to use (you can see the list of possible models [here](https://github.com/huggingface/transformers/blob/e6cff60b4cbc1158fbd6e4a1c3afda8dc224f566/examples/run_glue.py#L69)). It also supports using either the CPU, a single GPU, or multiple GPUs. It even supports using 16-bit precision if you want further speed up.\n",
        "\n",
        "Unfortunately, all of this configurability comes at the cost of *readability*. In this Notebook, we've simplified the code greatly and added plenty of comments to make it clear what's going on. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4JrUHXms16cn",
        "colab_type": "text"
      },
      "source": [
        "## 2.1. Download & Extract"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "36e94mPLXktE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "a=pd.read_csv('/content/train_set.csv')\n",
        "b=pd.read_csv('/content/train_labels.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IxN0sAOlb1em",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df=pd.concat([a,b], axis=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4SMZ5T5Imhlx",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "Let's extract the sentences and labels of our training set as numpy ndarrays."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GuE5BqICAne2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get the lists of sentences and their labels.\n",
        "sentences = df.Processed_tweets.values\n",
        "labels = df.Generalized_Hate.values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ex5O1eV-Pfct",
        "colab_type": "text"
      },
      "source": [
        "# 3. Tokenization & Input Formatting\n",
        "\n",
        "In this section, we'll transform our dataset into the format that BERT can be trained on."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8kEDRvShcU5",
        "colab_type": "text"
      },
      "source": [
        "## 3.1. BERT Tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bWOPOyWghJp2",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "To feed our text to BERT, it must be split into tokens, and then these tokens must be mapped to their index in the tokenizer vocabulary.\n",
        "\n",
        "The tokenization must be performed by the tokenizer included with BERT--the below cell will download this for us. We'll be using the \"uncased\" version here.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z474sSC6oe7A",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "a2ee6054-c879-48db-ac69-81ba30748104"
      },
      "source": [
        "from transformers import BertTokenizer\n",
        "\n",
        "# Load the BERT tokenizer.\n",
        "print('Loading BERT tokenizer...')\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading BERT tokenizer...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dFzmtleW6KmJ",
        "colab_type": "text"
      },
      "source": [
        "Let's apply the tokenizer to one sentence just to see the output.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dLIbudgfh6F0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 91
        },
        "outputId": "50eec0cd-5053-447c-94f6-3730361b4f59"
      },
      "source": [
        "# Print the original sentence.\n",
        "print(' Original: ', sentences[0])\n",
        "\n",
        "# Print the sentence split into tokens.\n",
        "print('Tokenized: ', tokenizer.tokenize(sentences[0]))\n",
        "\n",
        "# Print the sentence mapped to token ids.\n",
        "print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(sentences[0])))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " Original:  unsung hero rose schneiderman was a forerunner of the metoo movement history mondaymotivation URL URL\n",
            "Tokenized:  ['un', '##sun', '##g', 'hero', 'rose', 'schneider', '##man', 'was', 'a', 'forerunner', 'of', 'the', 'met', '##oo', 'movement', 'history', 'monday', '##mot', '##ivation', 'ur', '##l', 'ur', '##l']\n",
            "Token IDs:  [4895, 19729, 2290, 5394, 3123, 15159, 2386, 2001, 1037, 23993, 1997, 1996, 2777, 9541, 2929, 2381, 6928, 18938, 25761, 24471, 2140, 24471, 2140]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WeNIc4auFUdF",
        "colab_type": "text"
      },
      "source": [
        "When we actually convert all of our sentences, we'll use the `tokenize.encode` function to handle both steps, rather than calling `tokenize` and `convert_tokens_to_ids` separately. \n",
        "\n",
        "Before we can do that, though, we need to talk about some of BERT's formatting requirements."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "viKGCCh8izww",
        "colab_type": "text"
      },
      "source": [
        "## 3.2. Required Formatting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V6mceWWOjZnw",
        "colab_type": "text"
      },
      "source": [
        "### Special Tokens\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ykk0P9JiKtVe",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "**`[SEP]`**\n",
        "\n",
        "At the end of every sentence, we need to append the special `[SEP]` token. \n",
        "\n",
        "This token is an artifact of two-sentence tasks, where BERT is given two separate sentences and asked to determine something (e.g., can the answer to the question in sentence A be found in sentence B?). \n",
        "\n",
        "**`[CLS]`**\n",
        "\n",
        "For classification tasks, we must prepend the special `[CLS]` token to the beginning of every sentence."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u51v0kFxeteu",
        "colab_type": "text"
      },
      "source": [
        "### Sentence Length & Attention Mask\n",
        "\n",
        "\n",
        "BERT has two constraints:\n",
        "1. All sentences must be padded or truncated to a single, fixed length.\n",
        "2. The maximum sentence length is 512 tokens.\n",
        "\n",
        "Padding is done with a special `[PAD]` token, which is at index 0 in the BERT vocabulary. The below illustration demonstrates padding out to a \"MAX_LEN\" of 8 tokens.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l6w8elb-58GJ",
        "colab_type": "text"
      },
      "source": [
        "## 3.2. Sentences to IDs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1M296yz577fV",
        "colab_type": "text"
      },
      "source": [
        "The `tokenizer.encode` function combines multiple steps for us:\n",
        "1. Split the sentence into tokens.\n",
        "2. Add the special `[CLS]` and `[SEP]` tokens.\n",
        "3. Map the tokens to their IDs.\n",
        "\n",
        "Oddly, this function can perform truncating for us, but doesn't handle padding. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bBdb3pt8LuQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 73
        },
        "outputId": "f69d418c-948a-45c2-ad5f-b0941e66e371"
      },
      "source": [
        "# Tokenize all of the sentences and map the tokens to thier word IDs.\n",
        "input_ids = []\n",
        "\n",
        "# For every sentence...\n",
        "for sent in sentences:\n",
        "    # `encode` will:\n",
        "    #   (1) Tokenize the sentence.\n",
        "    #   (2) Prepend the `[CLS]` token to the start.\n",
        "    #   (3) Append the `[SEP]` token to the end.\n",
        "    #   (4) Map tokens to their IDs.\n",
        "    encoded_sent = tokenizer.encode(\n",
        "                        sent,                      # Sentence to encode.\n",
        "                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'\n",
        "\n",
        "                        # This function also supports truncation and conversion\n",
        "                        # to pytorch tensors, but we need to do padding, so we\n",
        "                        # can't use these features :( .\n",
        "                        #max_length = 128,          # Truncate all sentences.\n",
        "                        #return_tensors = 'pt',     # Return pytorch tensors.\n",
        "                   )\n",
        "    \n",
        "    # Add the encoded sentence to the list.\n",
        "    input_ids.append(encoded_sent)\n",
        "\n",
        "# Print sentence 0, now as a list of IDs.\n",
        "print('Original: ', sentences[0])\n",
        "print('Token IDs:', input_ids[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original:  unsung hero rose schneiderman was a forerunner of the metoo movement history mondaymotivation URL URL\n",
            "Token IDs: [101, 4895, 19729, 2290, 5394, 3123, 15159, 2386, 2001, 1037, 23993, 1997, 1996, 2777, 9541, 2929, 2381, 6928, 18938, 25761, 24471, 2140, 24471, 2140, 102]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WhwCKszh6ych",
        "colab_type": "text"
      },
      "source": [
        "## 3.3. Padding & Truncating"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xytsw1oIfnX0",
        "colab_type": "text"
      },
      "source": [
        "Pad and truncate our sequences so that they all have the same length, `MAX_LEN`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zqiWTDrn_nGB",
        "colab_type": "text"
      },
      "source": [
        "First, what's the maximum sentence length in our dataset?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JhUZO9vc_l6T",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "aebefa96-79a2-43de-ff7c-1f59ce7cb47d"
      },
      "source": [
        "print('Max sentence length: ', max([len(sen) for sen in input_ids]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Max sentence length:  198\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hp-54FcQ_p3h",
        "colab_type": "text"
      },
      "source": [
        "Given that, let's choose MAX_LEN = 195 and apply the padding.\n",
        "\n",
        "> we will use maxlen=192\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cp9BPRd1tMIo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 125
        },
        "outputId": "11fc3742-5086-4c62-921a-89bb1f1d76ff"
      },
      "source": [
        "# We'll borrow the `pad_sequences` utility function to do this.\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "# Set the maximum sequence length.\n",
        "# I've chosen 64 somewhat arbitrarily. It's slightly larger than the\n",
        "# maximum training sentence length of 47...\n",
        "MAX_LEN = 192\n",
        "\n",
        "print('\\nPadding/truncating all sentences to %d values...' % MAX_LEN)\n",
        "\n",
        "print('\\nPadding token: \"{:}\", ID: {:}'.format(tokenizer.pad_token, tokenizer.pad_token_id))\n",
        "\n",
        "# Pad our input tokens with value 0.\n",
        "# \"post\" indicates that we want to pad and truncate at the end of the sequence,\n",
        "# as opposed to the beginning.\n",
        "input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", \n",
        "                          value=0, truncating=\"post\", padding=\"post\")\n",
        "\n",
        "print('\\nDone.')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Padding/truncating all sentences to 192 values...\n",
            "\n",
            "Padding token: \"[PAD]\", ID: 0\n",
            "\n",
            "Done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kDs-MYtYH8sL",
        "colab_type": "text"
      },
      "source": [
        "## 3.4. Attention Masks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KhGulL1pExCT",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://)The attention mask simply makes it explicit which tokens are actual words versus which are padding. \n",
        "\n",
        "The BERT vocabulary does not use the ID 0, so if a token ID is 0, then it's padding, and otherwise it's a real token."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cDoC24LeEv3N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create attention masks\n",
        "attention_masks = []\n",
        "\n",
        "# For each sentence...\n",
        "for sent in input_ids:\n",
        "    \n",
        "    # Create the attention mask.\n",
        "    #   - If a token ID is 0, then it's padding, set the mask to 0.\n",
        "    #   - If a token ID is > 0, then it's a real token, set the mask to 1.\n",
        "    att_mask = [int(token_id > 0) for token_id in sent]\n",
        "    \n",
        "    # Store the attention mask for this sentence.\n",
        "    attention_masks.append(att_mask)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aRp4O7D295d_",
        "colab_type": "text"
      },
      "source": [
        "## 3.5. Training & Validation Split\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qu0ao7p8rb06",
        "colab_type": "text"
      },
      "source": [
        "Divide up our training set to use 90% for training and 10% for validation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aFbE-UHvsb7-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Use train_test_split to split our data into train and validation sets for\n",
        "# training\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Use 80% for training and 20% for test\n",
        "train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(input_ids, labels, \n",
        "                                                            random_state=42, test_size=0.2)\n",
        "# Do the same for the masks.\n",
        "train_masks, validation_masks, _, _ = train_test_split(attention_masks, labels,\n",
        "                                             random_state=42, test_size=0.2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7LzSbTqW9_BR",
        "colab_type": "text"
      },
      "source": [
        "## 3.6. Converting to PyTorch Data Types"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6p1uXczp-Je4",
        "colab_type": "text"
      },
      "source": [
        "Our model expects PyTorch tensors rather than numpy.ndarrays, so convert all of our dataset variables."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jw5K2A5Ko1RF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Convert all inputs and labels into torch tensors, the required datatype \n",
        "# for our model.\n",
        "train_inputs = torch.tensor(train_inputs)\n",
        "validation_inputs = torch.tensor(validation_inputs)\n",
        "\n",
        "train_labels = torch.tensor(train_labels)\n",
        "validation_labels = torch.tensor(validation_labels)\n",
        "\n",
        "train_masks = torch.tensor(train_masks)\n",
        "validation_masks = torch.tensor(validation_masks)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dD9i6Z2pG-sN",
        "colab_type": "text"
      },
      "source": [
        "We'll also create an iterator for our dataset using the torch DataLoader class. This helps save on memory during training because, unlike a for loop, with an iterator the entire dataset does not need to be loaded into memory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GEgLpFVlo1Z-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
        "\n",
        "# The DataLoader needs to know our batch size for training, so we specify it \n",
        "# here.\n",
        "# For fine-tuning BERT on a specific task, the authors recommend a batch size of\n",
        "# 16 or 32.\n",
        "\n",
        "batch_size = 32\n",
        "\n",
        "# Create the DataLoader for our training set.\n",
        "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n",
        "train_sampler = RandomSampler(train_data)\n",
        "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n",
        "\n",
        "# Create the DataLoader for our validation set.\n",
        "validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\n",
        "validation_sampler = SequentialSampler(validation_data)\n",
        "validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8bwa6Rts-02-",
        "colab_type": "text"
      },
      "source": [
        "# 4. Train Our Classification Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xYQ3iLO08SX",
        "colab_type": "text"
      },
      "source": [
        "Now that our input data is properly formatted, it's time to fine tune the BERT model. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D6TKgyUzPIQc",
        "colab_type": "text"
      },
      "source": [
        "## 4.1. BertForSequenceClassification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXYitPoE-cjH",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "We'll be using [BertForSequenceClassification](https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#bertforsequenceclassification). This is the normal BERT model with an added single linear layer on top for classification that we will use as a sentence classifier. As we feed input data, the entire pre-trained BERT model and the additional untrained classification layer is trained on our specific task. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gFsCTp_mporB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "929e8ccb-bc19-43dd-fb72-0845b15c17f9"
      },
      "source": [
        "from transformers import BertForSequenceClassification, AdamW, BertConfig\n",
        "\n",
        "# Load BertForSequenceClassification, the pretrained BERT model with a single \n",
        "# linear classification layer on top. \n",
        "model = BertForSequenceClassification.from_pretrained(\n",
        "    \"bert-base-uncased\", # Use the 12-layer BERT model, with an uncased vocab.\n",
        "    num_labels = 2, # The number of output labels--2 for binary classification.\n",
        "                    # You can increase this for multi-class tasks.   \n",
        "    output_attentions = False, # Whether the model returns attentions weights.\n",
        "    output_hidden_states = False, # Whether the model returns all hidden-states.\n",
        ")\n",
        "\n",
        "# Tell pytorch to run this model on the GPU.\n",
        "model.cuda()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
            "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
            "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BertForSequenceClassification(\n",
              "  (bert): BertModel(\n",
              "    (embeddings): BertEmbeddings(\n",
              "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
              "      (position_embeddings): Embedding(512, 768)\n",
              "      (token_type_embeddings): Embedding(2, 768)\n",
              "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "      (dropout): Dropout(p=0.1, inplace=False)\n",
              "    )\n",
              "    (encoder): BertEncoder(\n",
              "      (layer): ModuleList(\n",
              "        (0): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (1): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (2): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (3): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (4): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (5): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (6): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (7): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (8): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (9): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (10): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (11): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "    (pooler): BertPooler(\n",
              "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "      (activation): Tanh()\n",
              "    )\n",
              "  )\n",
              "  (dropout): Dropout(p=0.1, inplace=False)\n",
              "  (classifier): Linear(in_features=768, out_features=2, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 88
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8PIiVlDYCtSq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 647
        },
        "outputId": "956d28eb-a363-4367-c73f-0eabc27ce77e"
      },
      "source": [
        "# Get all of the model's parameters as a list of tuples.\n",
        "params = list(model.named_parameters())\n",
        "\n",
        "print('The BERT model has {:} different named parameters.\\n'.format(len(params)))\n",
        "\n",
        "print('==== Embedding Layer ====\\n')\n",
        "\n",
        "for p in params[0:5]:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
        "\n",
        "print('\\n==== First Transformer ====\\n')\n",
        "\n",
        "for p in params[5:21]:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
        "\n",
        "print('\\n==== Output Layer ====\\n')\n",
        "\n",
        "for p in params[-4:]:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The BERT model has 201 different named parameters.\n",
            "\n",
            "==== Embedding Layer ====\n",
            "\n",
            "bert.embeddings.word_embeddings.weight                  (30522, 768)\n",
            "bert.embeddings.position_embeddings.weight                (512, 768)\n",
            "bert.embeddings.token_type_embeddings.weight                (2, 768)\n",
            "bert.embeddings.LayerNorm.weight                              (768,)\n",
            "bert.embeddings.LayerNorm.bias                                (768,)\n",
            "\n",
            "==== First Transformer ====\n",
            "\n",
            "bert.encoder.layer.0.attention.self.query.weight          (768, 768)\n",
            "bert.encoder.layer.0.attention.self.query.bias                (768,)\n",
            "bert.encoder.layer.0.attention.self.key.weight            (768, 768)\n",
            "bert.encoder.layer.0.attention.self.key.bias                  (768,)\n",
            "bert.encoder.layer.0.attention.self.value.weight          (768, 768)\n",
            "bert.encoder.layer.0.attention.self.value.bias                (768,)\n",
            "bert.encoder.layer.0.attention.output.dense.weight        (768, 768)\n",
            "bert.encoder.layer.0.attention.output.dense.bias              (768,)\n",
            "bert.encoder.layer.0.attention.output.LayerNorm.weight        (768,)\n",
            "bert.encoder.layer.0.attention.output.LayerNorm.bias          (768,)\n",
            "bert.encoder.layer.0.intermediate.dense.weight           (3072, 768)\n",
            "bert.encoder.layer.0.intermediate.dense.bias                 (3072,)\n",
            "bert.encoder.layer.0.output.dense.weight                 (768, 3072)\n",
            "bert.encoder.layer.0.output.dense.bias                        (768,)\n",
            "bert.encoder.layer.0.output.LayerNorm.weight                  (768,)\n",
            "bert.encoder.layer.0.output.LayerNorm.bias                    (768,)\n",
            "\n",
            "==== Output Layer ====\n",
            "\n",
            "bert.pooler.dense.weight                                  (768, 768)\n",
            "bert.pooler.dense.bias                                        (768,)\n",
            "classifier.weight                                           (2, 768)\n",
            "classifier.bias                                                 (2,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qRWT-D4U_Pvx",
        "colab_type": "text"
      },
      "source": [
        "## 4.2. Optimizer & Learning Rate Scheduler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8o-VEBobKwHk",
        "colab_type": "text"
      },
      "source": [
        "Now that we have our model loaded we need to grab the training hyperparameters from within the stored model.\n",
        "\n",
        "For the purposes of fine-tuning, the authors recommend choosing from the following values:\n",
        "- Batch size: 16, 32  (We chose 32 when creating our DataLoaders).\n",
        "- Learning rate (Adam): 5e-5, 3e-5, 2e-5  (We'll use 2e-5).\n",
        "- Number of epochs: 2, 3, 4  (We'll use 4).\n",
        "\n",
        "The epsilon parameter `eps = 1e-8` is \"a very small number to prevent any division by zero in the implementation\" (from [here](https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/)).\n",
        "\n",
        "You can find the creation of the AdamW optimizer in `run_glue.py` [here](https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L109)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GLs72DuMODJO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Note: AdamW is a class from the huggingface library (as opposed to pytorch) \n",
        "# I believe the 'W' stands for 'Weight Decay fix\"\n",
        "optimizer = AdamW(model.parameters(),\n",
        "                  lr = 2e-5, # args.learning_rate - default is 5e-5, our notebook had 2e-5\n",
        "                  eps = 1e-8 # args.adam_epsilon  - default is 1e-8.\n",
        "                )\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-p0upAhhRiIx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from transformers import get_linear_schedule_with_warmup\n",
        "\n",
        "# Number of training epochs (authors recommend between 2 and 4)\n",
        "epochs = 3\n",
        "\n",
        "# Total number of training steps is number of batches * number of epochs.\n",
        "total_steps = len(train_dataloader) * epochs\n",
        "\n",
        "# Create the learning rate scheduler.\n",
        "scheduler = get_linear_schedule_with_warmup(optimizer, \n",
        "                                            num_warmup_steps = 0, # Default value in run_glue.py\n",
        "                                            num_training_steps = total_steps)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RqfmWwUR_Sox",
        "colab_type": "text"
      },
      "source": [
        "## 4.3. Training Loop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QXZhFb4LnV5",
        "colab_type": "text"
      },
      "source": [
        "Below is our training loop. There's a lot going on, but fundamentally for each pass in our loop we have a trianing phase and a validation phase. At each pass we need to:\n",
        "\n",
        "Training loop:\n",
        "- Unpack our data inputs and labels\n",
        "- Load data onto the GPU for acceleration\n",
        "- Clear out the gradients calculated in the previous pass. \n",
        "    - In pytorch the gradients accumulate by default (useful for things like RNNs) unless you explicitly clear them out.\n",
        "- Forward pass (feed input data through the network)\n",
        "- Backward pass (backpropagation)\n",
        "- Tell the network to update parameters with optimizer.step()\n",
        "- Track variables for monitoring progress\n",
        "\n",
        "Evalution loop:\n",
        "- Unpack our data inputs and labels\n",
        "- Load data onto the GPU for acceleration\n",
        "- Forward pass (feed input data through the network)\n",
        "- Compute loss on our validation data and track variables for monitoring progress\n",
        "\n",
        "So please read carefully through the comments to get an understanding of what's happening. If you're unfamiliar with pytorch a quick look at some of their [beginner tutorials](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) will help show you that training loops really involve only a few simple steps; the rest is usually just decoration and logging.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pE5B99H5H2-W",
        "colab_type": "text"
      },
      "source": [
        "Define a helper function for calculating accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9cQNvaZ9bnyy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# Function to calculate the accuracy of our predictions vs labels\n",
        "def flat_accuracy(preds, labels):\n",
        "    pred_flat = np.argmax(preds, axis=1).flatten()\n",
        "    labels_flat = labels.flatten()\n",
        "    return np.sum(pred_flat == labels_flat) / len(labels_flat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNhRtWPXH9C3",
        "colab_type": "text"
      },
      "source": [
        "Helper function for formatting elapsed times.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gpt6tR83keZD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import time\n",
        "import datetime\n",
        "\n",
        "def format_time(elapsed):\n",
        "    '''\n",
        "    Takes a time in seconds and returns a string hh:mm:ss\n",
        "    '''\n",
        "    # Round to the nearest second.\n",
        "    elapsed_rounded = int(round((elapsed)))\n",
        "    \n",
        "    # Format as hh:mm:ss\n",
        "    return str(datetime.timedelta(seconds=elapsed_rounded))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cfNIhN19te3N",
        "colab_type": "text"
      },
      "source": [
        "We're ready to kick off the training!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6J-FYdx6nFE_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 755
        },
        "outputId": "c7eb4d46-ac42-40be-e714-5300b1097700"
      },
      "source": [
        "import random\n",
        "\n",
        "# This training code is based on the `run_glue.py` script here:\n",
        "# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128\n",
        "\n",
        "\n",
        "# Set the seed value all over the place to make this reproducible.\n",
        "seed_val = 42\n",
        "\n",
        "random.seed(seed_val)\n",
        "np.random.seed(seed_val)\n",
        "torch.manual_seed(seed_val)\n",
        "torch.cuda.manual_seed_all(seed_val)\n",
        "\n",
        "# Store the average loss after each epoch so we can plot them.\n",
        "loss_values = []\n",
        "\n",
        "# For each epoch...\n",
        "for epoch_i in range(0, epochs):\n",
        "    \n",
        "    # ========================================\n",
        "    #               Training\n",
        "    # ========================================\n",
        "    \n",
        "    # Perform one full pass over the training set.\n",
        "\n",
        "    print(\"\")\n",
        "    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))\n",
        "    print('Training...')\n",
        "\n",
        "    # Measure how long the training epoch takes.\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Reset the total loss for this epoch.\n",
        "    total_loss = 0\n",
        "\n",
        "    # Put the model into training mode. Don't be mislead--the call to \n",
        "    # `train` just changes the *mode*, it doesn't *perform* the training.\n",
        "    # `dropout` and `batchnorm` layers behave differently during training\n",
        "    # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)\n",
        "    model.train()\n",
        "\n",
        "    # For each batch of training data...\n",
        "    for step, batch in enumerate(train_dataloader):\n",
        "\n",
        "        # Progress update every 40 batches.\n",
        "        if step % 40 == 0 and not step == 0:\n",
        "            # Calculate elapsed time in minutes.\n",
        "            elapsed = format_time(time.time() - t0)\n",
        "            \n",
        "            # Report progress.\n",
        "            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))\n",
        "\n",
        "        # Unpack this training batch from our dataloader. \n",
        "        #\n",
        "        # As we unpack the batch, we'll also copy each tensor to the GPU using the \n",
        "        # `to` method.\n",
        "        #\n",
        "        # `batch` contains three pytorch tensors:\n",
        "        #   [0]: input ids \n",
        "        #   [1]: attention masks\n",
        "        #   [2]: labels \n",
        "        b_input_ids = batch[0].to(device)\n",
        "        b_input_mask = batch[1].to(device)\n",
        "        b_labels = batch[2].to(device)\n",
        "\n",
        "        # Always clear any previously calculated gradients before performing a\n",
        "        # backward pass. PyTorch doesn't do this automatically because \n",
        "        # accumulating the gradients is \"convenient while training RNNs\". \n",
        "        # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)\n",
        "        model.zero_grad()        \n",
        "\n",
        "        # Perform a forward pass (evaluate the model on this training batch).\n",
        "        # This will return the loss (rather than the model output) because we\n",
        "        # have provided the `labels`.\n",
        "        # The documentation for this `model` function is here: \n",
        "        # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification\n",
        "        outputs = model(b_input_ids, \n",
        "                    token_type_ids=None, \n",
        "                    attention_mask=b_input_mask, \n",
        "                    labels=b_labels)\n",
        "        \n",
        "        # The call to `model` always returns a tuple, so we need to pull the \n",
        "        # loss value out of the tuple.\n",
        "        loss = outputs[0]\n",
        "\n",
        "        # Accumulate the training loss over all of the batches so that we can\n",
        "        # calculate the average loss at the end. `loss` is a Tensor containing a\n",
        "        # single value; the `.item()` function just returns the Python value \n",
        "        # from the tensor.\n",
        "        total_loss += loss.item()\n",
        "\n",
        "        # Perform a backward pass to calculate the gradients.\n",
        "        loss.backward()\n",
        "\n",
        "        # Clip the norm of the gradients to 1.0.\n",
        "        # This is to help prevent the \"exploding gradients\" problem.\n",
        "        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
        "\n",
        "        # Update parameters and take a step using the computed gradient.\n",
        "        # The optimizer dictates the \"update rule\"--how the parameters are\n",
        "        # modified based on their gradients, the learning rate, etc.\n",
        "        optimizer.step()\n",
        "\n",
        "        # Update the learning rate.\n",
        "        scheduler.step()\n",
        "\n",
        "    # Calculate the average loss over the training data.\n",
        "    avg_train_loss = total_loss / len(train_dataloader)            \n",
        "    \n",
        "    # Store the loss value for plotting the learning curve.\n",
        "    loss_values.append(avg_train_loss)\n",
        "\n",
        "    print(\"\")\n",
        "    print(\"  Average training loss: {0:.2f}\".format(avg_train_loss))\n",
        "    print(\"  Training epoch took: {:}\".format(format_time(time.time() - t0)))\n",
        "        \n",
        "    # ========================================\n",
        "    #               Validation\n",
        "    # ========================================\n",
        "    # After the completion of each training epoch, measure our performance on\n",
        "    # our validation set.\n",
        "\n",
        "    print(\"\")\n",
        "    print(\"Running Validation...\")\n",
        "\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Put the model in evaluation mode--the dropout layers behave differently\n",
        "    # during evaluation.\n",
        "    model.eval()\n",
        "\n",
        "    # Tracking variables \n",
        "    eval_loss, eval_accuracy = 0, 0\n",
        "    nb_eval_steps, nb_eval_examples = 0, 0\n",
        "\n",
        "    # Evaluate data for one epoch\n",
        "    for batch in validation_dataloader:\n",
        "        \n",
        "        # Add batch to GPU\n",
        "        batch = tuple(t.to(device) for t in batch)\n",
        "        \n",
        "        # Unpack the inputs from our dataloader\n",
        "        b_input_ids, b_input_mask, b_labels = batch\n",
        "        \n",
        "        # Telling the model not to compute or store gradients, saving memory and\n",
        "        # speeding up validation\n",
        "        with torch.no_grad():        \n",
        "\n",
        "            # Forward pass, calculate logit predictions.\n",
        "            # This will return the logits rather than the loss because we have\n",
        "            # not provided labels.\n",
        "            # token_type_ids is the same as the \"segment ids\", which \n",
        "            # differentiates sentence 1 and 2 in 2-sentence tasks.\n",
        "            # The documentation for this `model` function is here: \n",
        "            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification\n",
        "            outputs = model(b_input_ids, \n",
        "                            token_type_ids=None, \n",
        "                            attention_mask=b_input_mask)\n",
        "        \n",
        "        # Get the \"logits\" output by the model. The \"logits\" are the output\n",
        "        # values prior to applying an activation function like the softmax.\n",
        "        logits = outputs[0]\n",
        "\n",
        "        # Move logits and labels to CPU\n",
        "        logits = logits.detach().cpu().numpy()\n",
        "        label_ids = b_labels.to('cpu').numpy()\n",
        "        \n",
        "        # Calculate the accuracy for this batch of test sentences.\n",
        "        tmp_eval_accuracy = flat_accuracy(logits, label_ids)\n",
        "        \n",
        "        # Accumulate the total accuracy.\n",
        "        eval_accuracy += tmp_eval_accuracy\n",
        "\n",
        "        # Track the number of batches\n",
        "        nb_eval_steps += 1\n",
        "\n",
        "    # Report the final accuracy for this validation run.\n",
        "    print(\"  Accuracy: {0:.2f}\".format(eval_accuracy/nb_eval_steps))\n",
        "    print(\"  Validation took: {:}\".format(format_time(time.time() - t0)))\n",
        "\n",
        "print(\"\")\n",
        "print(\"Training complete!\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "======== Epoch 1 / 3 ========\n",
            "Training...\n",
            "  Batch    40  of    138.    Elapsed: 0:01:10.\n",
            "  Batch    80  of    138.    Elapsed: 0:02:21.\n",
            "  Batch   120  of    138.    Elapsed: 0:03:31.\n",
            "\n",
            "  Average training loss: 0.15\n",
            "  Training epoch took: 0:04:02\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.97\n",
            "  Validation took: 0:00:23\n",
            "\n",
            "======== Epoch 2 / 3 ========\n",
            "Training...\n",
            "  Batch    40  of    138.    Elapsed: 0:01:10.\n",
            "  Batch    80  of    138.    Elapsed: 0:02:21.\n",
            "  Batch   120  of    138.    Elapsed: 0:03:31.\n",
            "\n",
            "  Average training loss: 0.14\n",
            "  Training epoch took: 0:04:02\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.97\n",
            "  Validation took: 0:00:23\n",
            "\n",
            "======== Epoch 3 / 3 ========\n",
            "Training...\n",
            "  Batch    40  of    138.    Elapsed: 0:01:11.\n",
            "  Batch    80  of    138.    Elapsed: 0:02:21.\n",
            "  Batch   120  of    138.    Elapsed: 0:03:31.\n",
            "\n",
            "  Average training loss: 0.13\n",
            "  Training epoch took: 0:04:02\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.97\n",
            "  Validation took: 0:00:23\n",
            "\n",
            "Training complete!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-G03mmwH3aI",
        "colab_type": "text"
      },
      "source": [
        "Let's take a look at our training loss over all batches:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68xreA9JAmG5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 483
        },
        "outputId": "b0d5f1a2-f3c8-4481-972b-0bf2126826f2"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "% matplotlib inline\n",
        "\n",
        "import seaborn as sns\n",
        "\n",
        "# Use plot styling from seaborn.\n",
        "sns.set(style='darkgrid')\n",
        "\n",
        "# Increase the plot size and font size.\n",
        "sns.set(font_scale=1.5)\n",
        "plt.rcParams[\"figure.figsize\"] = (12,6)\n",
        "\n",
        "# Plot the learning curve.\n",
        "plt.plot(loss_values, 'b-o')\n",
        "\n",
        "# Label the plot.\n",
        "plt.title(\"Training loss\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Loss\")\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwUAAAGaCAYAAABeyu/GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVhV1f4/8Pc5cJgHkUlBQJwOKoNMjlSGaYQ4kTikImo03Pzem+UNzdIyzavgVRrMr4iiaCEYRAXOKXXNFNREE1FRkSMOKMooHODw+6Of53tPDLIF3Azv1/P4PJ2191rr4+fhST5n7bW2pLa2thZERERERNRpScUOgIiIiIiIxMWigIiIiIiok2NRQERERETUybEoICIiIiLq5FgUEBERERF1ciwKiIiIiIg6ORYFRETUbAqFAnK5HJ9//vkTj7Fo0SLI5fIWjOrJyOVyLFq0SOwwiIieKm2xAyAiopYn5JfrQ4cOoUePHq0YDRERtXUSvryMiKjjSU5O1vh88uRJ7Nq1C1OnToWnp6fGtdGjR8PAwKBZ89XW1kKpVEJLSwva2k/2fVNVVRVUKhV0dXWbFUtzyeVyTJo0Cf/6179EjYOI6GniSgERUQc0YcIEjc81NTXYtWsXBg0aVOfaX5WWlsLIyEjQfBKJpNm/zMtksmb1JyKiJ8c9BUREnZivry9mzZqF8+fPY968efD09MT48eMB/FkcrFu3DkFBQRgyZAicnZ0xevRoRERE4OHDhxrj1Len4L/bDh8+jJdffhkuLi7w8fHB6tWrUV1drTFGfXsKHrWVlJRg2bJlGDZsGFxcXDBt2jScOXOmzt/n/v37WLx4MYYMGQJ3d3cEBwfj/PnzmDVrFnx9fZuVq4SEBEyaNAmurq7w9PTE3LlzkZGRUee+I0eOYObMmRgyZAhcXV0xcuRIzJ8/H1evXlXfc/PmTSxevBjPP/88nJ2dMWzYMEybNg1JSUnNipGI6ElxpYCIqJPLz8/H7Nmz4efnhzFjxqC8vBwAcPv2bezevRtjxoxBQEAAtLW1ceLECWzevBlZWVmIjo5u0vhpaWn4+uuvMW3aNLz88ss4dOgQtmzZAlNTU7zxxhtNGmPevHno2rUr3nrrLTx48ABbt27Fa6+9hkOHDqlXNZRKJebMmYOsrCwEBgbCxcUF2dnZmDNnDkxNTZ8sOf9feHg4Nm/eDFdXV7zzzjsoLS1FfHw8Zs+ejQ0bNuC5554DAJw4cQJvvvkm+vbti9dffx3Gxsa4c+cOjh07huvXr8PR0RHV1dWYM2cObt++jVdeeQU9e/ZEaWkpsrOzkZGRgUmTJjUrViKiJ8GigIiok1MoFFixYgWCgoI02u3s7HDkyBGNx3pmzJiB9evX46uvvkJmZiZcXV0fO/7ly5fx448/qjczT58+HePGjcOOHTuaXBQMGDAAH330kfpz79698fbbb+PHH3/EtGnTAPz5TX5WVhbefvttvPnmm+p7+/Xrh+XLl8PW1rZJc/3VlStXEB0dDQ8PD2zbtg06OjoAgKCgIIwdOxYff/wxDhw4AC0tLRw6dAgqlQpbt26Fubm5eoy33npLIx9Xr17FwoULERoa+kQxERG1ND4+RETUyXXp0gWBgYF12nV0dNQFQXV1NYqKilBYWIjhw4cDQL2P79Rn1KhRGqcbSSQSDBkyBAUFBSgrK2vSGCEhIRqfhw4dCgDIzc1Vtx0+fBhaWloIDg7WuDcoKAjGxsZNmqc+hw4dQm1tLV599VV1QQAA1tbWCAwMxI0bN3D+/HkAUM+zb9++Oo9HPfLonuPHj+PevXtPHBcRUUviSgERUSdnZ2cHLS2teq/t3LkTcXFxuHz5MlQqlca1oqKiJo//V126dAEAPHjwAIaGhoLHMDMzU/d/RKFQwMrKqs54Ojo66NGjB4qLi5sU718pFAoAQN++fetce9SWl5cHFxcXzJgxA4cOHcLHH3+MiIgIeHp64plnnkFAQAC6du0KALC1tcUbb7yBTZs2wcfHB/3798fQoUPh5+fXpJUXIqLWwJUCIqJOTl9fv972rVu3Yvny5bCyssLy5cuxadMmbN26VX1UZ1NPtG6o4GiJMdraqdpmZmbYvXs3tm/fjlmzZqGsrAyrVq3Ciy++iNOnT6vvW7BgAfbv34/3338fdnZ22L17N4KCghAeHi5i9ETUmXGlgIiI6pWcnAxbW1tERUVBKv2/75B+/vlnEaNqmK2tLY4dO4aysjKN1YKqqiooFAqYmJg80biPVikuXboEe3t7jWuXL1/WuAf4s4AZMmQIhgwZAgC4cOECXn75ZXz11VfYtGmTxrizZs3CrFmzUFlZiXnz5mHz5s2YO3euxn4EIqKngSsFRERUL6lUColEovFtfHV1NaKiokSMqmG+vr6oqanB9u3bNdrj4+NRUlLSrHElEgmio6NRVVWlbr9z5w4SExNha2uLAQMGAAAKCwvr9O/Vqxd0dXXVj1uVlJRojAMAurq66NWrF4CmP5ZFRNSSuFJARET18vPzw9q1axEaGorRo0ejtLQUP/744xO/sbi1BQUFIS4uDuvXr8f169fVR5Lu3bsXDg4ODW78fZxevXqpv8WfOXMmXnrpJZSVlSE+Ph7l5eWIiIhQP9704Ycf4tatW/Dx8YGNjQ0qKiqwZ88elJWVqV8ad/z4cXz44YcYM2YMHB0dYWhoiHPnzmH37t1wc3NTFwdERE9T2/w/OxERiW7evHmora3F7t27sXLlSlhaWuKll17Cyy+/DH9/f7HDq0NHRwfbtm3DmjVrcOjQIezZsweurq6IiYnBkiVLUFFR8cRj//Of/4SDgwO+/vprrF27FjKZDG5ubli7di28vLzU902YMAGJiYlISkpCYWEhjIyM0KdPH3z22Wd48cUXAQByuRyjR4/GiRMn8MMPP0ClUqF79+54/fXXMXfu3GbngYjoSUhq29ouLSIiohZUU1ODoUOHwtXVtckvXCMi6my4p4CIiDqM+lYD4uLiUFxcjBEjRogQERFR+8DHh4iIqMP44IMPoFQq4e7uDh0dHZw+fRo//vgjHBwcMGXKFLHDIyJqs/j4EBERdRjfffcddu7ciWvXrqG8vBzm5uZ47rnn8I9//AMWFhZih0dE1GaxKCAiIiIi6uS4p4CIiIiIqJNjUUBERERE1Mlxo3EbcP9+GVSqp/8Ul7m5Ee7dK33q87ZXzJcwzJcwzJcwzJcwzJcwzJcwzJcwYuVLKpXAzMywwessCtoAlapWlKLg0dzUdMyXMMyXMMyXMMyXMMyXMMyXMMyXMG0xX3x8iIiIiIiok2NRQERERETUybEoICIiIiLq5FgUEBERERF1ciwKiIiIiIg6ORYFRERERESdHIsCIiIiIqJOjkUBEREREVEnx6KAiIiIiKiT4xuNO6Fjf9xCYloOCosr0dVEF4HP9cawgd3EDouIiIiIRMKioJM59sctbNtzAcpqFQDgXnEltu25AAAsDIiIiIg6KT4+1MkkpuWoC4JHlNUqJKbliBQREREREYlN1JUCpVKJyMhIJCcno7i4GE5OTliwYAGGDRvWaL/MzEwkJiYiMzMTFy9eRFVVFbKzs+vcp1AoMGrUqHrHiIqKwrPPPgsAUKlUSEpKwoEDB5CVlYWioiL06NEDAQEBmDt3LnR0dASP2VbdK64U1E5EREREHZ+oRcGiRYuwf/9+BAcHw8HBAUlJSQgNDUVsbCzc3d0b7JeWloaEhATI5XLY2dnhypUrjc4zfvx4+Pj4aLQ5OTmp//vhw4d4//33MWjQIEybNg3m5uY4ffo0IiMj8dtvvyEmJkbwmG2VuYluvQVAFyNdEaIhIiIiorZAtKIgMzMTKSkpWLx4MUJCQgAAEydOREBAACIiIrBz584G+06fPh2hoaHQ09PDypUrH1sUDBw4EBMmTGjwukwmwzfffAMPDw9125QpU2Bra4vPP/8cx48fx5AhQwSN2VYFPtdbY0/BIw8rq5B9/T7k9mYiRUZEREREYhFtT8HevXshk8kQFBSkbtPV1cXkyZNx8uRJ3Llzp8G+FhYW0NPTEzRfeXk5lEplvdd0dHQ0CoJHRo8eDQDIyan/efvGxmyrhg3shtkvOcHcRBcS/LlyEPR8b5gZ6yH8m99xID0PtbW1YodJRERERE+RaCsFWVlZcHR0hKGhoUa7q6sramtrkZWVBSsrqxaZKzIyEqtWrYJEIoGbmxsWLlwIb2/vx/a7e/cuAMDMrO635086ZlswbGA3DBvYDZaWxigoKAEAjBxki80/nsc3hy7h6q1izPZzgq5MS+RIiYiIiOhpEK0oKCgogLW1dZ12S0tLAGh0paCppFIpfHx8MHr0aFhZWSE3NxfR0dGYM2cOYmJi4OXl1Wj/zZs3w9jYWGPvQHPHbKv0dbXxVqALUn69hu9+uYr8gjLMD3SBRRd9sUMjIiIiolYmWlFQUVEBmUxWp11X988Nr5WVzT8Nx8bGBtHR0Rpt/v7+GDt2LCIiIhAXF9dg340bN+LXX3/F8uXLYWxs3CJjNsTc3Ehwn5ZiaWms8XnuRFe4yq0RsfMkPtmegX/O9IK7vGVWbDqCv+aLGsd8CcN8CcN8CcN8CcN8CcN8CdMW8yVaUaCnp4eqqqo67Y+KgUfFQUuztrbG2LFjER8fj4cPH0Jfv+434ampqVi/fj2mTp2KqVOntsiYjbl3rxQq1dN/jv+/Hx/6bw4WBvgg2BNfJJ7FsqhjmPxcb/gNsYdEInnqMbYlDeWL6sd8CcN8CcN8CcN8CcN8CcN8CSNWvqRSSaNfRIu20djS0rLeR4QKCgoAoMX2E9Sne/fuUKlUKC4urnPt6NGjeO+99/D8889j2bJlLTJme2RtZoAlszzhJbdCwpEcfJX8ByqU1WKHRUREREStQLSiwMnJCVevXkVZWZlG+5kzZ9TXW0teXh60tLRgampaZ+758+fDxcUF69atg5ZW0zfaNjRme6ano403JgzElOf74GT2HazcfhK375eLHRYRERERtTDRigI/Pz9UVVUhISFB3aZUKpGYmAgPDw/1JuT8/PwGjwR9nMLCwjptubm5SElJgZeXl8axpjk5OXjttddga2uLjRs3NnjkqZAxOwKJRAK/IfZ4Z+ogFJUpsTwmA2cu3xU7LCIiIiJqQaLtKXBzc4Ofnx8iIiJQUFAAe3t7JCUlIT8/H6tWrVLfFxYWhhMnTiA7O1vdduPGDSQnJwMAzp49CwDYsGEDgD9XGHx9fQEA4eHhyMvLw9ChQ2FlZYXr16+rNwKHhYWpxystLcW8efNQXFyMefPm4ciRIxqxyuVy9cpFU8fsaAb27Iqls73wRdJZfLY7ExN8HBEwoieknXyfAREREVFHIFpRAABr1qzB+vXrkZycjKKiIsjlcmzatAmenp6N9lMoFIiMjNRoe/R50qRJ6qJgxIgRiIuLw44dO1BSUgITExOMGDEC8+fPR9++fdV9Hzx4gJs3bwIA1q5dW2e++fPnq4uCpo7ZEVl00cf7Mz2xbW82vvvPVVy7VYJXAwbAQE/UHyMiIiIiaiZJLV9fK7q2dvrQ49TW1uLQSQV2/XQZFl308T+BLrCxMHx8x3aOpysIw3wJw3wJw3wJw3wJw3wJw3wJw9OHqMOQSCR4wcsOC6cNwsOKKnyyPQMns5v/sjkiIiIiEgeLAnpicnszLA3xho25Ib5MOodv03JEWfEgIiIiouZhUUDN0tVED4tmeOBZt+5IOZaL9bvPoKyi7kvpiIiIiKjtYlFAzSbTliLkpf4I9pMj69p9LI9JR96dUrHDIiIiIqImYlFALWbkIFuEzfBAVbUKK2MzcPz8bbFDIiIiIqImYFFALaqPrSmWhXjDwdoY//v9H9j10yXUqFRih0VEREREjWBRQC3O1EgX/5zujlEePbDvRB7+vesMisuVYodFRERERA1gUUCtQltLihlj+mHe2P64pCjCJzHpuHarWOywiIiIiKgeLAqoVY1w6Y73Z3kAAD6NPYWjZ2+KHBERERER/RWLAmp1PbuZ4MMQb/TtYYrolCzs3H8R1TXcZ0BERETUVrAooKfCxEAH70x1w4uD7XDolALh35xGUWml2GEREREREVgU0FOkJZViqm9fvD5+IHJvleDjmHTk3CgSOywiIiKiTo9FAT11QwZYY0mwF2TaUvxr5ykc+f2G2CERERERdWosCkgUdlZG+HC2N/o7mGH73mzE7MlCVTX3GRARERGJgUUBicZIX4a3g9wwdpgDfj5zE6u/PoXC4gqxwyIiIiLqdFgUkKikUglefq433prkjBt3y7A8Jh3Z1++LHRYRERFRp8KigNoET7kVPgj2gr6eDBFxv+NgRh5qa2vFDouIiIioU2BRQG2GrYUhPgz2gksvc3x98BI2/5gFZVWN2GERERERdXgsCqhNMdDTxvyXXTDxGUf89sctfLrjJO4+eCh2WEREREQdGosCanOkEgnGj3DE3ye7ouBBBZZvy8D5a4Vih0VERETUYbEooDbLrY8Fls72gqmhDtbu+h17judynwERERFRK2BRQG2adVcDLAn2hGc/SyQczsHG5D9QqeQ+AyIiIqKWxKKA2jw9HW28OdEZQSN7IyP7DlbEZuD2/XKxwyIiIiLqMFgUULsgkUjw0lAHvDNlEB6UVOKTmAxk5twTOywiIiKiDkHUokCpVCI8PBw+Pj5wdXXFlClTcOzYscf2y8zMxEcffYTAwEA4OztDLpfXe59CoYBcLq/3z88//1zn/pycHMybNw/u7u4YPHgwwsLCUFhYd4OrSqVCVFQUfH194eLignHjxiE1NVV4AkiwgY5dsTTEGxameohMOIMfjl6FivsMiIiIiJpFW8zJFy1ahP379yM4OBgODg5ISkpCaGgoYmNj4e7u3mC/tLQ0JCQkQC6Xw87ODleuXGl0nvHjx8PHx0ejzcnJSePzrVu3MGPGDJiYmGDBggUoLy/Hli1bcPHiRcTHx0Mmk6nvXbduHTZt2oSpU6fC2dkZhw4dwoIFCyCVSuHn5/cEmSAhLLvoY/EsT2zbewFJv1zFtVsleDVgAPR1Rf1xJiIiImq3RPstKjMzEykpKVi8eDFCQkIAABMnTkRAQAAiIiKwc+fOBvtOnz4doaGh0NPTw8qVKx9bFAwcOBATJkxo9J6NGzeisrISsbGxsLa2BgC4urpizpw5SE5OxuTJkwEAt2/fxtatWxEcHIwlS5YAAIKCgjBz5kysWbMGY8aMgVTKp7Jam65MC6EBA+DYzQS7frqMFdszMD/QBd3NDcUOjYiIiKjdEe23171790ImkyEoKEjdpquri8mTJ+PkyZO4c+dOg30tLCygp6cnaL7y8nIolcoGr+/fvx++vr7qggAAhg8fjp49e2LPnj3qtoMHD6KqqgqvvPKKuk0ikWD69Om4ceMGMjMzBcVFT04ikWC0tx0WThuE0odV+GRbBk5fLBA7LCIiIqJ2R7SiICsrC46OjjA01Pxm19XVFbW1tcjKymqxuSIjI+Hu7g5XV1dMnToV6enpGtdv376Ne/fuwdnZuU5fV1dXjViysrJgZGQER0fHOvcBwPnz51ssbmoaJwczLAvxRndzA3yeeBaJP1+BSsV9BkRERERNJdrjQwUFBRrfyj9iaWkJAI2uFDSVVCqFj48PRo8eDSsrK+Tm5iI6Ohpz5sxBTEwMvLy8NOZ6NPdf47l37x5qamqgpaWFgoICWFhYtGrcJFxXEz0smuGB2P0X8eOv15B7qwSvjR8AQz3Z4zsTERERdXKiFQUVFRUam3cf0dXVBQBUVlY2ew4bGxtER0drtPn7+2Ps2LGIiIhAXFycxlw6OjoNxlNRUQFDQ0NUVFQ0et+TxG1ubiS4T0uxtDQWbe7W8F6wN1yPXcOm787i09hTWDJnMBy6m7TY+B0tX62N+RKG+RKG+RKG+RKG+RKG+RKmLeZLtKJAT08PVVVVddof/VL96JfslmZtbY2xY8ciPj4eDx8+hL6+vnqu+vYcPIrn0R4GPT29Ru97krjv3SsV5XEXS0tjFBSUPPV5W5tXXwt0me6BL787i3ci0zDXvz8G96+7KiVUR81Xa2G+hGG+hGG+hGG+hGG+hGG+hBErX1KppNEvokXbU2BpaVnvozYFBX9uFLWysmq1ubt37w6VSoXi4mKNuR7N/dd4zM3NoaWlBeDPuO/evVvvff89FomrTw9TLAvxhr2VMTYm/4H4w5dRo1KJHRYRERFRmyRaUeDk5ISrV6+irKxMo/3MmTPq660lLy8PWlpaMDU1BfDn6kHXrl1x7ty5OvdmZmaif//+6s/9+/dHaWkprl69Wm/c/30viauLkS7ee8Udz3vYYu/x6/j3rjMoKW/4BCoiIiKizkq0osDPzw9VVVVISEhQtymVSiQmJsLDw0O9CTk/Px85OTlPNEd9byPOzc1FSkoKvLy8NI41HTNmDH766Sfcvn1b3Xbs2DFcu3ZN44Vko0aNgkwmw9dff61uq62tRVxcHGxsbODm5vZEsVLr0NaSYtYYOeb4O+GSogjLYzKQe4tLnERERET/TbQ9BW5ubvDz80NERAQKCgpgb2+PpKQk5OfnY9WqVer7wsLCcOLECWRnZ6vbbty4geTkZADA2bNnAQAbNmwA8OcKg6+vLwAgPDwceXl5GDp0KKysrHD9+nX15uKwsDCNeN544w3s3bsXwcHBmDlzJsrLyxEdHQ0nJyeNF59169YNwcHB2LJlCyorK+Hi4oKDBw8iIyMD69at44vL2qhnXG3Qw9IIXyadxac7TmK2nxzDnbuLHRYRERFRmyBaUQAAa9aswfr165GcnIyioiLI5XJs2rQJnp6ejfZTKBSIjIzUaHv0edKkSeqiYMSIEYiLi8OOHTtQUlICExMTjBgxAvPnz0ffvn01+nfv3h07duzAv/71L6xduxYymQwjR47E4sWL65w2tHDhQpiammLXrl1ITEyEo6Mj1q5dC39//+amhFqRY3cTLJ3tjY3J57D5xyxcu1mCKb59oK3FQo6IiIg6N0ltbS3f8iQynj70dNWoVEg4nIP96XnoZ9cFb050hqlh3WNm/6qz5utJMV/CMF/CMF/CMF/CMF/CMF/C8PQhojZCSyrFtFF98dq4Abh2sxjLY9KRk18kdlhEREREomFRQJ3W0IHd8P4sT2hJJVi98xR+PpMvdkhEREREomBRQJ2avbUxloZ4Q25vhpg9F7Bt7wVUVfN9BkRERNS5sCigTs9IX4YFQW7wH+qAtN/zsebrU7hfUil2WERERERPDYsCIvy5+WbyyN7420RnKArK8HFMOi7mPRA7LCIiIqKngkUB0X/xcrLCB8Ge0NfRQvg3p3HopAI8oIuIiIg6OhYFRH9ha2mED2d7wdmxK3YeuIgtKVlQVtWIHRYRERFRq2FRQFQPAz0Z/meyK8aP6Imj525h1c5TuFNYLnZYRERERK2CRQFRA6QSCSY+0wt/f9kVd+6XY8H6NGRdKxQ7LCIiIqIWx6KA6DEG9bXAh7O9YWqkg4hdv2PfievcZ0BEREQdCosCoibo1tUAEX9/Fh59LbHrp8v43+//QKWS+wyIiIioY2BRQNREBnoy/G2SM15+rhfSs+5gZexJ3LnPfQZERETU/rEoIBJAIpFg7LCeWDDFDfdLKrA8JgNnr9wTOywiIiKiZmFRQPQEnHuZ48MQb3Q10cP6+DP48ddr3GdARERE7RaLAqInZNVFH0tmeWLwAGsk/nwFXyadw8PKarHDIiIiIhKMRQFRM+jqaOG1cQMwzbcPfr90Fyu2Z+DmvTKxwyIiIiIShEUBUTNJJBKMGWyPd6cNQkl5FVZsz8DpSwVih0VERETUZCwKiFpIfwczLAvxhpWZAT7/9iy+++UKVNxnQERERO0AiwKiFmRuqofFMzwwwqUbvj96DZ/tzkR5RZXYYRERERE1ikUBUQvTkWlhrn9/zBrTD39cLcTybRm4UVAqdlhEREREDWJRQNQKJBIJnvfogX9Od0elsgYrtp9ExoU7YodFREREVC8WBUStqJ9dFywN8UYPK0Ns+O4cEo5chkrFfQZERETUtrAoIGplZsa6eG+6B0a622LPb9exLv53lD7kPgMiIiJqO1gUED0FMm0pgl+UI+QlJ2TnPcDymHRcv10idlhEREREAFgUED1Vz7rZYNEMT9SoavFp7Ekc++OW2CERERERQVvMyZVKJSIjI5GcnIzi4mI4OTlhwYIFGDZsWKP9MjMzkZiYiMzMTFy8eBFVVVXIzs5+7HypqalYsGABjI2NkZGRoXFNLpc32G/48OHYunUrAEChUGDUqFH13hcVFYVnn332sXFQ59bLxgRLQ7zx1XfnEPXDeVy9WYwpz/eBthZrdCIiIhKHqEXBokWLsH//fgQHB8PBwQFJSUkIDQ1FbGws3N3dG+yXlpaGhIQEyOVy2NnZ4cqVK4+dq6KiAuHh4TAwMKj3+po1a+q0nTt3Dtu3b8eIESPqXBs/fjx8fHw02pycnB4bBxEAmBrqYOG0QYg/fBkHMxTIu12KNyc6w8RQR+zQiIiIqBMSrSjIzMxESkoKFi9ejJCQEADAxIkTERAQgIiICOzcubPBvtOnT0doaCj09PSwcuXKJhUFUVFR0NHRga+vL9LS0upcnzBhQp22EydOQCKRICAgoM61gQMH1tuHqKm0taR45YV+cOxmgpi9F/BxTDrmB7rAsbuJ2KERERFRJyPa8wp79+6FTCZDUFCQuk1XVxeTJ0/GyZMncedOw2e6W1hYQE9Pr8lz5efnY/PmzQgLC4NMJmtSH6VSif3798Pb2xvdunWr957y8nIolcomx0FUn2HO3fD+TE9IJRKs2nEKv5zJFzskIiIi6mREKwqysrLg6OgIQ0NDjXZXV1fU1tYiKyurxeZavXo13N3d4evr2+Q+aWlpKC4uxvjx4+u9HhkZCXd3d7i6umLq1KlIT09vqXCpE3LoZoxlc7zRz84UW/dcwPZ92aiuUYkdFhEREXUSoj0+VFBQAGtr6zrtlpaWANDoSoEQJ06cwIEDB5CYmCio3w8//AAdHR28+OKLGu1SqRQ+Pj4YPXo0rKyskJubi+joaMyZMwcxMTHw8vJqkbip8zHSl+GdKYPw7c6Fi9kAACAASURBVM852PPbdSju/LnPwMxYV+zQiIiIqIMTrSioqKio91EeXd0/fwGqrKxs9hw1NTVYsWIFAgMDBW0CLi0txZEjR/Dcc8/BxETz+W4bGxtER0drtPn7+2Ps2LGIiIhAXFyc4DjNzY0E92kplpbGos3dHj2NfP0tyB0ufa3w2a7TWLE9A4tme2OAo3mrz9sa+PMlDPMlDPMlDPMlDPMlDPMlTFvMl2hFgZ6eHqqq6r7V9VEx8Kg4aI5du3ZBoVBgy5Ytgvrt27cPlZWVGDduXJPut7a2xtixYxEfH4+HDx9CX19f0Hz37pVCpaoV1KclWFoao6CAL9BqqqeZLydbE7w/yxNfJJ7F+xuOYvoLffG8uy0kEslTmb8l8OdLGOZLGOZLGOZLGOZLGOZLGLHyJZVKGv0iWrQ9BZaWlvU+IlRQUAAAsLKyatb4SqUSn332GQIDA1FRUQGFQgGFQoHy8nKoVCooFAoUFhbW2/eHH36AsbExnn/++SbP1717d6hUKhQXFzcrbqJHelgaYelsLwx07Iod+y9ia+oFVFXXiB0WERERdUCirRQ4OTkhNjYWZWVlGpuNz5w5o77eHBUVFbh//z5iY2MRGxtb5/qoUaPg7++PdevWabTfuXMHx48fx6RJk6Cj0/Qz4/Py8qClpQVTU9NmxU303wz0ZPj7ZFd8/5+r+P7oNSgKSjE/0AVdTZp++hYRERHR44hWFPj5+WHLli1ISEhQv6dAqVQiMTERHh4e6k3I+fn5ePjwIXr37i1ofH19fXz55Zd12rdv347MzExERETUu9E5NTUVKpWqwUeHCgsL0bVrV4223NxcpKSkwMvLS9BRqURNIZVIMPGZXnCwNkbUj+fxcUw63pzgDCcHM7FDIyIiog5CtKLAzc0Nfn5+iIiIQEFBAezt7ZGUlIT8/HysWrVKfV9YWBhOnDiB7OxsdduNGzeQnJwMADh79iwAYMOGDQD+XGHw9fWFTCbDCy+8UGfegwcP4vz58/VeA4Dvv/8eVlZWGDJkSL3Xw8PDkZeXh6FDh8LKygrXr19Xby4OCwt7gkwQNY17P0t8ONsLXySeRUTc75jyfG+M9rZrV/sMiIiIqG0SrSgAgDVr1mD9+vVITk5GUVER5HI5Nm3aBE9Pz0b7KRQKREZGarQ9+jxp0iRB7yP4b1euXMEff/yBOXPmQCqtf7vFiBEjEBcXhx07dqCkpAQmJiYYMWIE5s+fj759+z7RvERN1d3cEB8EeyE6JQtxP13GtVslmP2SE3RlWmKHRkRERO2YpLa29ukfe0MaePpQ+9CW8qWqrUXqsVwk/XwFPayMMD/QBZZdhJ161draUr7aA+ZLGOZLGOZLGOZLGOZLGJ4+REQtRiqRIGB4T/wjyA33iiqwPCYd567eEzssIiIiaqdYFBC1Y669zbE0xAtmxrpYt+sMUo5dAxf/iIiISCgWBUTtnJWZAZbM8oJ3fyt8m3YFX313Dg8rq8UOi4iIiNoRUTcaE1HL0NXRwuvjB6JnNxMkHLmM/Hvl+J9AF1h3NRA7NCIiImoHuFJA1EFIJBL4DbHHu1MHobhMieXb0vH75btih0VERETtAIsCog5mQM+uWBriBasuBvhsdyaS/3MVKu4zICIiokawKCDqgCxM9bF4pgeGO3dD8n+u4otvz6K8gvsMiIiIqH4sCog6KB2ZFuaN7Y8Zo/vh7JV7+GR7Bm7cLRM7LCIiImqDWBQQdWASiQSjPHvgn9Pd8bCyGiu2ZyDjwh2xwyIiIqI2hkUBUSfQz64LloV4w9bCEBu+O4dv03JEeYs2ERERtU0sCog6CTNjXYS94oHnBtkg5Vgu1iecQenDKrHDIiIiojaARQFRJyLTlmK2nxNm+8lx4fp9LI9Jx/XbJWKHRURERCJjUUDUCT03yBZhMzxQo6rFp7En8dsft8QOiYiIiETEooCok+ptY4qlId7o2c0Ym344j7hDl1CjUokdFhEREYmARQFRJ2ZqqIOF090xyrMH9qfnYW3c7yguV4odFhERET1lLAqIOjltLSlmjO6HeWP7Iye/GMtj0nH1ZrHYYREREdFTxKKAiAAAI1y64/2ZnpAAWLXjFP6TeVPskIiIiOgpYVFARGoO3YyxNMQbfXuYYktqFmL3Z6O6hvsMiIiIOjoWBUSkwdhAB+9MdYPfYHscPnUDa745jQellWKHRURERK2IRQER1aEllWKKbx+8MWEgrt8uwccx6bh8o0jssIiIiKiVsCggogYN7m+ND2Z5QUdbitU7T+HI6Ruora0VOywiIiJqYSwKiKhRPayMsDTEGwN6dsX2fdmI2XMBVdU1YodFRERELYhFARE9lqGeDP+Y7IqA4Q74JfMm/rXzNAqLK8QOi4iIiFoIiwIiahKpVILAZ3tjfqALbt4rw8cx6ci+fl/ssIiIiKgFiFoUKJVKhIeHw8fHB66urpgyZQqOHTv22H6ZmZn46KOPEBgYCGdnZ8jl8ibNl5qaCrlcDi8vrzrXFi1aBLlcXufPlClT6tyrUqkQFRUFX19fuLi4YNy4cUhNTW1SDETtnUc/S3wQ7AVDPRnCv/kdBzLyuM+AiIiondMWc/JFixZh//79CA4OhoODA5KSkhAaGorY2Fi4u7s32C8tLQ0JCQmQy+Wws7PDlStXHjtXRUUFwsPDYWBg0OA9+vr6+PjjjzXaunbtWue+devWYdOmTZg6dSqcnZ1x6NAhLFiwAFKpFH5+fo+Nhai9s7EwxIezvbD5x/P45uAlXLtZjGA/J+jKtMQOjYiIiJ6AaEVBZmYmUlJSsHjxYoSEhAAAJk6ciICAAERERGDnzp0N9p0+fTpCQ0Ohp6eHlStXNqkoiIqKgo6ODnx9fZGWllbvPdra2pgwYUKj49y+fRtbt25FcHAwlixZAgAICgrCzJkzsWbNGowZMwZSKZ/Koo5PX1cbbwW6IOXXa/jul6u4UVCG+YEusOiiL3ZoREREJJBov73u3bsXMpkMQUFB6jZdXV1MnjwZJ0+exJ07dxrsa2FhAT09vSbPlZ+fj82bNyMsLAwymazRe2tqalBaWtrg9YMHD6KqqgqvvPKKuk0ikWD69Om4ceMGMjMzmxwXUXsnlUgwboQj/hHkioKiCnwck44/rhaKHRYREREJJFpRkJWVBUdHRxgaGmq0u7q6ora2FllZWS021+rVq+Hu7g5fX99G7ysrK4Onpyc8PT0xZMgQrFq1CpWVmm9yzcrKgpGRERwdHevEDQDnz59vsbiJ2gvX3hZYGuKFLsa6+Hf879jzWy73GRAREbUjoj0+VFBQAGtr6zrtlpaWANDoSoEQJ06cwIEDB5CYmNjofZaWlnj11VfRv39/qFQqHD58GDExMcjJycHmzZs14rawsGj1uInaG2szAyyZ5YmtqReQcCQH+fcfYsaoPtDTEXXrEhERETWBaP9aV1RU1Psoj66uLgDU+Yb+SdTU1GDFihUIDAyEk5NTo/e+++67Gp8DAgJgbW2N6OhoHD16FCNGjFDHraOj06Jxm5sbCe7TUiwtjUWbuz1ivh7vw1eHIunIZWxLOY+82yVYMmcwbCzE+xlvT/jzJQzzJQzzJQzzJQzzJUxbzJdoRYGenh6qqqrqtD/6pfrRL9nNsWvXLigUCmzZsuWJ+s+dOxfR0dE4duyYuijQ09ODUqmsc29z4r53rxQq1dN/1MLS0hgFBSVPfd72ivlqumecu6GXrSn+tS0db/87Da+NGwC3PnVX2Oj/8OdLGOZLGOZLGOZLGOZLGLHyJZVKGv0iWrQ9BZaWlvU+alNQUAAAsLKyatb4SqUSn332GQIDA1FRUQGFQgGFQoHy8nKoVCooFAoUFja+IdLCwgIymQxFRUUacd+9e7fV4ibqKAb1s8KyEG9Ymurhs92Z+P7oVai4z4CIiKhNEq0ocHJywtWrV1FWVqbRfubMGfX15qioqMD9+/cRGxuLUaNGqf/s27cPZWVlGDVqFD755JNGx7h16xaqqqo03lXQv39/lJaW4urVq/XG3b9//2bFTdSRWHTRx+JZnhg60Brf/XIVX3x7FuUV1WKHRURERH8hWlHg5+eHqqoqJCQkqNuUSiUSExPh4eGh3oScn5+PnJwcwePr6+vjyy+/rPNnyJAh6mtz584F8OejP/UdQ7phwwYAgI+Pj7pt1KhRkMlk+Prrr9VttbW1iIuLg42NDdzc3ATHStSR6cq08GrAALzyQl9k5tzDiu0ZyL9b9viORERE9NSItqfAzc0Nfn5+iIiIQEFBAezt7ZGUlIT8/HysWrVKfV9YWBhOnDiB7OxsdduNGzeQnJwMADh79iyA//sF3snJCb6+vpDJZHjhhRfqzHvw4EGcP39e41pBQQEmTZqEgIAA9OrVS3360LFjx+Dv7w9vb2/1vd26dUNwcDC2bNmCyspKuLi44ODBg8jIyMC6dev44jKiekgkErzgZQc7KyN89d05fLI9A6+OHQBPuaXYoRERERFELAoAYM2aNVi/fj2Sk5NRVFQEuVyOTZs2wdPTs9F+CoUCkZGRGm2PPk+aNOmx7yP4KxMTE4wcORJHjx5FUlISVCoVevbsiUWLFiE4OLjO/QsXLoSpqSl27dqFxMREODo6Yu3atfD39xc0L1FnI7c3w9IQb3yZdA5fJp3F2GEOmPRML0ilErFDIyIi6tQktXzDkOh4+lD7wHwJ01i+qqprsPPARfx85iace3XF6+MHwlCv8beNd3T8+RKG+RKG+RKG+RKG+RKGpw8REf1/Mm0thLzUH8F+cmRdu4/lMenIu1N3Xw8RERE9HSwKiEg0IwfZImyGB5TVKqyMzcDx87fFDomIiKhTYlFARKLqY2uKZSHesLc2xv9+/wd2/XQJNSqV2GERERF1Ki1SFFRXV2Pfvn2Ij49Xv8SLiKipuhjp4r3p7vD1sMW+E3n4964zKCmv++ZwIiIiah2CTx9as2YNjh8/jm+//RbAn2f0z5kzBxkZGaitrUWXLl0QHx8Pe3v7Fg+WiDoubS0pZo6Ro2c3E2zfl43lMemYH+gKh27GYodGRETU4QleKfjll1/g5eWl/vzTTz8hPT0d8+bNw9q1awEAmzZtarkIiahT8XHtjsUzPVAL4NMdJ3H07E2xQyIiIurwBK8U3Lp1Cw4ODurPhw8fRo8ePbBw4UIAwKVLl/DDDz+0XIRE1Ok4djfB0hBvbPzuHKJTsnDtZgmmjuoDbS1ugyIiImoNgv+Fraqqgrb2/9USx48fx/Dhw9Wf7ezsuK+AiJrNxEAH704bhBcH2+HQKQUivjmNotJKscMiIiLqkAQXBd26dcPp06cB/LkqkJeXB29vb/X1e/fuwcDAoOUiJKJOS0sqxVTfvnht/ABcu1WCj2PSkXOjSOywiIiIOhzBjw+NHTsWGzZsQGFhIS5dugQjIyM899xz6utZWVncZExELWrogG6wMTfEF4lnsfrrU3hldD+MHGQrdlhEREQdhuCVgtdffx2TJk3C77//DolEgtWrV8PExAQAUFJSgp9++gnDhg1r8UCJqHOztzbG0hBvONmbYfvebMTsuYCqar7PgIiIqCUIXinQ0dHBp59+Wu81Q0ND/Oc//4Genl6zAyMi+isjfRneDnJD0i9XkHIsF4qCUrw1yQVmxrpih0ZERNSutehRHtXV1TA2NoZMJmvJYYmI1KRSCV5+rjfemuSMG3fL8HFMOi7mPRA7LCIionZNcFGQlpaGzz//XKNt586d8PDwwKBBg/Duu++iqqqqxQIkIqqPp9wKHwR7QV9XG+HfnMbBjDzU1taKHRYREVG7JLgoiI6OxpUrV9Sfc3Jy8Omnn8LKygrDhw9Hamoqdu7c2aJBEhHVx9bCEB8Ge8Gllzm+PngJ0SlZUFbViB0WERFRuyO4KLhy5QqcnZ3Vn1NTU6Grq4vdu3dj8+bN8Pf3x3fffdeiQRIRNcRATxvzX3bBRB9H/HruFlbtOIW7RQ/FDouIiKhdEVwUFBUVwczMTP35119/xdChQ2FkZAQAGDx4MBQKRctFSET0GFKJBON9HPH3ya6486Acy2MycP5aodhhERERtRuCiwIzMzPk5+cDAEpLS3H27Fl4eXmpr1dXV6Omhsv3RPT0DepjgaWzvWFiqIO1u37H3uPXuc+AiIioCQQfSTpo0CDExcWhT58++Pnnn1FTU4Nnn31WfT03NxdWVlYtGiQRUVNZdzXAB8Ge2JKShfjDl3HtVjHmvNQfujpaYodGRETUZgleKfj73/8OlUqFt99+G4mJiZg4cSL69OkDAKitrcXBgwfh4eHR4oESETWVno423pzojMkjeyP9wh2sjM3AnfvlYodFRETUZgleKejTpw9SU1Nx6tQpGBsbw9vbW32tuLgYs2fPxpAhQ1o0SCIioSQSCfyHOsDe2gj/m/wHlsdk4LXxA+Ha21zs0IiIiNqcJ3p5WZcuXeDr66tREACAqakpZs+eDScnpxYJjoiouZwdzbE0xBvmpnqITDiDH369BhX3GRAREWkQvFLwyPXr13Ho0CHk5eUBAOzs7DBq1CjY29u3WHBERC3Bsos+3p/liW17LyDp5yu4drMYrwYMgL7uE/8vkIiIqEN5on8R169fj6ioqDqnDIWHh+P111/HP/7xjxYJjoiopejKtBAaMAA9u5kg/qfLWLE9A/MDXdDd3FDs0IiIiEQnuCjYvXs3Nm7cCHd3d7z66qvo27cvAODSpUuIjo7Gxo0bYWdnh8DAwBYPloioOSQSCcZ428HeyghfJZ/DJ9syEBowAO79LMUOjYiISFSSWoGHeAcGBkImk2Hnzp3Q1tasKaqrqzFjxgxUVVUhMTHxsWMplUpERkYiOTkZxcXFcHJywoIFCzBs2LBG+2VmZiIxMRGZmZm4ePEiqqqqkJ2d/dj5UlNTsWDBAhgbGyMjI0PdrlKpkJSUhAMHDiArKwtFRUXo0aMHAgICMHfuXOjo6KjvVSgUGDVqVL3jR0VFaRzP2lT37pVCpXr6zzhbWhqjoKDkqc/bXjFfwrT1fBUWV+CLxLO4dqsE44b3xIRnHCGVSESLp63nq61hvoRhvoRhvoRhvoQRK19SqQTm5kYNXhe8UpCTk4N33nmnTkEAANra2vD398e///3vJo21aNEi7N+/H8HBwXBwcEBSUhJCQ0MRGxsLd3f3BvulpaUhISEBcrkcdnZ2uHLlymPnqqioQHh4OAwMDOpce/jwId5//30MGjQI06ZNg7m5OU6fPo3IyEj89ttviImJqdNn/Pjx8PHx0WjjBmui9qOriR4Wz/RA7L6L+OHXa8i9XYLQcQNgqCcTOzQiIqKnTnBRIJPJUF7e8HnfZWVlkMke/49qZmYmUlJSsHjxYoSEhAAAJk6ciICAAERERGDnzp0N9p0+fTpCQ0Ohp6eHlStXNqkoiIqKgo6ODnx9fZGWllbn7/TNN99ovF9hypQpsLW1xeeff47jx4/XOWZ14MCBmDBhwmPnJaK2S6athTn+TnDsboyvD17CJzEZmP+yC3pYNvxNChERUUck+EhSFxcX7Nq1C3fv3q1z7d69e4iPj4ebm9tjx9m7dy9kMhmCgoLUbbq6upg8eTJOnjyJO3fuNNjXwsICenp6TY45Pz8fmzdvRlhYWL0Fi46OTr0vXBs9ejSAP1dH6lNeXg6lUtnkOIio7ZFIJHjeowfCXvFAZVUNVmzPwIms22KHRURE9FQJLgr+9re/oaCgAP7+/li9ejW+/fZbfPvtt1i9ejX8/f1x9+5dvPnmm48dJysrC46OjjA01Dz5w9XVFbW1tcjKyhIaWoNWr14Nd3d3+Pr6Cur3qPAxMzOrcy0yMhLu7u5wdXXF1KlTkZ6e3iKxEpE4+vQwxbI53rC3MsbG5D+QcPgyalQqscMiIiJ6KgQ/PuTt7Y3PP/8cn3zyCbZu3apxzcbGBqtXr4aXl9djxykoKIC1tXWddkvLP08BaWylQIgTJ07gwIEDTdr4/FebN2+GsbGxxt4BqVQKHx8fjB49GlZWVsjNzUV0dDTmzJmDmJiYJv3diaht6mKki/deccc3By9hz/HryL1dgtfHD4Sxgc7jOxMREbVjT/SeAl9fX4wcORLnzp2DQqEA8OfLywYOHIj4+Hj4+/sjNTW10TEqKirqfZRHV1cXAFBZWfkkoWmoqanBihUrEBgYKHgT8MaNG/Hrr79i+fLlMDY2Vrfb2NggOjpa415/f3+MHTsWERERiIuLExxnYzvBW5ulpfHjbyI15kuY9pqvd2Z6waWvJb5KzMTK2JN4P2Qwevfo0urzttd8iYX5Eob5Eob5Eob5EqYt5uuJX+cplUrh6uoKV1dXjfb79+/j6tWrj+2vp6eHqqqqOu2PioFHxUFz7Nq1CwqFAlu2bBHULzU1FevXr8fUqVMxderUx95vbW2NsWPHIj4+Hg8fPoS+vr6g+XgkafvAfAnT3vM1qFdXLJrhgS8Sz+Kfn/+C2X5yDHfu3mrztfd8PW3MlzDMlzDMlzDMlzBt9UhSwXsKWoqlpWW9jwgVFBQAAKysrJo1vlKpxGeffYbAwEBUVFRAoVBAoVCgvLwcKpUKCoUChYWFdfodPXoU7733Hp5//nksW7asyfN1794dKpUKxcXFzYqbiNoOx+4mWBbijV7dTbD5xyx8feAiqmu4z4CIiDqeJ14paC4nJyfExsairKxMY7PxmTNn1Nebo6KiAvfv30dsbCxiY2PrXB81ahT8/f2xbt06jbnnz58PFxcXrFu3DlpaWk2eLy8vD1paWjA1NW1W3ETUtpgY6uDdaYOQcDgHBzLycP1OKd6c6AxTQ+4zICKijkO0osDPzw9btmxBQkKC+j0FSqUSiYmJ8PDwUG9Czs/Px8OHD9G7d29B4+vr6+PLL7+s0759+3ZkZmYiIiJCY6NzTk4OXnvtNdja2mLjxo0NHnlaWFiIrl27arTl5uYiJSUFXl5ego5KJaL2QVtLiukv9EXP7sbYtucClsek461JLuhlYyJ2aERERC1CtKLAzc0Nfn5+iIiIQEFBAezt7ZGUlIT8/HysWrVKfV9YWBhOnDiB7OxsdduNGzeQnJwMADh79iwAYMOGDQD+XGHw9fWFTCbDCy+8UGfegwcP4vz58xrXSktLMW/ePBQXF2PevHk4cuSIRh+5XK5euQgPD0deXh6GDh0KKysrXL9+Xb25OCwsrAUyQ0Rt1bCB3WBrYYgvEs/iXztPYuYYOZ51sxE7LCIiomZrUlHw16NHG3Pq1Kkm37tmzRqsX78eycnJKCoqglwux6ZNm+Dp6dloP4VCgcjISI22R58nTZok+H0EDx48wM2bNwEAa9eurXN9/vz56qJgxIgRiIuLw44dO1BSUgITExOMGDEC8+fPR9++fQXNS0Ttj721MZaGeON/k88hZs8FXLtZjOkv9INMW7QtWkRERM0mqa2tfeyxN0Kf75dIJC368rGOjqcPtQ/MlzAdPV8qVS0Sf76C1N9y0dvGBH+b5AIz4yc/Na2j56ulMV/CMF/CMF/CMF/CtNXTh5q0UrB9+/YWC4iIqCOQSiWYPLI3enYzRnRKFj6OScffJjqjn13rv8+AiIiopTWpKBg8eHBrx0FE1C55OVmhu7kBvkg8i/BvTmP6C33xvLstJBKJ2KERERE1GR+CJSJqJltLI3w42wsDHbtix/6L2JKaBWVVjdhhERERNRmLAiKiFmCgJ8PfJ7ti/IieOHr2FlbtPIV7RRVih0VERNQkLAqIiFqIVCLBxGd64X9edsGd++X4OCYdWbn3xQ6LiIjosVgUEBG1MPe+lvgg2AvGBjKsjfsd+05cRxMOeiMiIhINiwIiolbQ3dwQHwR7wb2vBXb9dBn/+/0fqFRynwEREbVNLAqIiFqJvq42/jbJGS8/1wvpWXewMvYk7jx4KHZYREREdbAoICJqRRKJBGOH9cTbU9xwv6QCn8Sk49yVe2KHRUREpIFFARHRU+DSyxwfhnjDzFgP6+LPIOXYNe4zICKiNoNFARHRU2LVRR9LZnli8ABrfJt2BRuSzuFhZbXYYREREbEoICJ6mnR1tPDauAGY5tsHpy/dxYrtGbh5r0zssIiIqJNjUUBE9JRJJBKMGWyPd6cNQkl5FVZsz8DxczfFDouIiDoxFgVERCLp72CGZSHesDIzwIqtJ/DdL1eg4j4DIiISAYsCIiIRmZvqYfEMD4zytsP3R6/h892ZKK+oEjssIiLqZFgUEBGJTEemhX9MdcfMMf1w7mohPtmWgRsFpWKHRUREnQiLAiKiNkAikcDXowf+Od0dD5U1WLH9JDIu3BE7LCIi6iRYFBARtSH97LpgWYg3elgaYsN355Bw5DJUKu4zICKi1sWigIiojTEz1sV7r3hg5CAb7PntOtYlnEHpQ+4zICKi1sOigIioDZJpSxHs54SQl5yQff0+lsek4/rtErHDIiKiDopFARFRG/asmw3CZnigRlWLT2NP4tgft8QOiYiIOiAWBUREbVxvG1MsDfFGz+4miPrhPL45eAnVNSqxwyIiog6ERQERUTtgaqiDhdMG4QWvHjiQkYe1cb+juEwpdlhERNRBsCggImontLWkeOWFfggNGIArN4vxcUw6rt4sFjssIiLqAEQtCpRKJcLDw+Hj4wNXV1dMmTIFx44de2y/zMxMfPTRRwgMDISzszPkcnmT5ktNTYVcLoeXl1e913NycjBv3jy4u7tj8ODBCAsLQ2FhYZ37VCoVoqKi4OvrCxcXF4wbNw6pqalNioGIqLmGOXfD+zM9IZVIsGrHKfxyJl/skIiIqJ0TtShYtGgRtm3bhvHjx2PJkiWQSqUIDQ3F6dOnG+2XlpaGhIQEAICdnV2T5qqoqEB4eDgMDAzqvX7r1i3MmDEDeXl5WLBgAebOnYvDhw9j3rx5qKrSPApw3bp1iIiIgI+PDz788EPY2NhgwYIF2Lt3b5NiISJqLoduxlga4oV+KR8ZLAAAIABJREFUdqbYuucCYvdlc58BERE9MdGKgszMTKSkpGDhwoV47733MHXqVGzbtg3du3dHREREo32nT5+OkydPIjExET4+Pk2aLyoqCjo6OvD19a33+saNG1FZWYnY2FgEBwfjjTfewPr163H+/HkkJyer77t9+za2bt2K4OD/196dhzdV5f8DfydtkrZJF9omXegKQlu2AgVZVaCCtYAIPxyGrYrCgA7zHXH5AjJ+HXHBQRxARh8FigyMslTpIKhsgqiAIIst0AJaCrSUNqGlLd2StMnvjza3jUmBdEvTvF/Pw+Pk3HNzTz5zCPeTe5YkLF26FH/4wx/w0UcfYcCAAVi+fDkMBv6jTERtw9NDigV/iMWjg8Jw6Mx1LP/sDIrLtPZuFhEROSC7JQV79uyBRCLBE088IZTJZDJMnjwZp06dglqtbvRcf39/uLm53fO18vLysH79eixcuBASicRqnX379mHUqFEICAgQyoYOHYqIiAh88803QtmBAweg1+sxbdo0oUwkEmHq1Km4fv060tPT77ldRETN5SIW44mR92HehJ64pr6N1zf+jN9yS+zdLCIicjB2SwoyMzMRGRkJuVxuVt6nTx8YjUZkZma22LX+8Y9/oF+/fo0+JSgoKEBhYSF69eplcaxPnz5mbcnMzIRCoUBkZKRFPQDIyMhosXYTEd2r+2MC8LekAZC5uuAfn53GodO5MBqN9m4WERE5CLslBRqNBiqVyqJcqVQCwB2fFNjixIkT2L9/PxYtWtRoHdO1TNf+fXsKCwtRU1MDoLbd/v7+Vus1fC8iorYWolTg1acGoGekLzbvu4RPvrkAfXWNvZtFREQOwNVeF66qqrI6lEcmkwEAtNrmj4utqanBm2++iUmTJiE6OrrReqZrSaXSRttTVVUFuVyOqqqqO9ZrSrv9/BQ2n9NSlEpPu13bETFetmG8bNMS8VICeGPeMHy27wK27b+EgluVWPzk/VB2cm9+A9sZ9i/bMF62Ybxsw3jZpj3Gy25JgZubm8WqPkD9TbXpJrs5tm3bhtzcXGzYsOGO9UzX0uksNwIytcc0h8HNze2O9ZrS7sLCMhgMbf+YX6n0hEZzu82v66gYL9swXrZp6Xg9EhcCpacM63dn4K//PIRnJ/RCdHinFnt/e2P/sg3jZRvGyzaMl23sFS+xWHTHH6LtNnxIqVRaHWqj0WgAwOrQIlvodDq8//77mDRpEqqqqpCbm4vc3FxUVFTAYDAgNzdX2IPAdC3TtX/fHj8/P7i4uAjtvnnzZqu1m4iopfTvrsSrTw6Awl2CFVt/wb6fczjPgIiIrLJbUhAdHY3s7GyUl5eblaelpQnHm6Oqqgq3bt3C5s2bER8fL/zZu3cvysvLER8fjzfeeAMAEBAQAF9fX5w7d87ifdLT0xETEyO8jomJQVlZGbKzs622u2FdIiJ7C/KT429JAxB7nx+2fvsr1u3KgFbPeQZERGTObklBQkIC9Hq9sAkZUPvr/o4dO9C/f39hadC8vDxkZWXZ/P7u7u744IMPLP4MGjRIOPb0008L9ceMGYODBw+ioKBAKDt27BiuXLmChIQEoSw+Ph4SiQSfffaZUGY0GrF161YEBwcjNjbW5rYSEbUmd5kr/jypNyY+2AXHMwqwbPMpaIor7d0sIiJqR+w2pyA2NhYJCQlYsWIFNBoNwsLCkJqairy8PCxbtkyot3DhQpw4cQIXL14Uyq5fvy5sKHb27FkAwIcffgig9gnDqFGjIJFI8PDDD1tc98CBA8jIyLA4Nm/ePOzZswdJSUmYMWMGKioqkJycjOjoaEyYMEGoFxgYiKSkJGzYsAFarRa9e/fGgQMHcPLkSaxcuRJisV03iSYiskosEmH80AiEB3hi7ZfnsXTjz5g7oSd6RfrZu2lERNQO2C0pAIDly5dj1apV2LlzJ0pKShAVFYW1a9ciLi7ujufl5uZi9erVZmWm1xMnTmx0P4I7CQoKwn/+8x+88847eO+99yCRSDBixAgsXrzYYrWhl156Cd7e3ti2bRt27NiByMhIvPfee0hMTLT5ukREbalPVz+8+tQAfLDjLFZuT8P/e6grHh0UBpFIZO+mERGRHYmMnHVmd1x9yDEwXrZhvGzT1vHS6mrwyTeZOJGpxoAoJWYlxsBdZtffiWzC/mUbxss2jJdtGC/btNfVhxznXwAiImoxMqkL5j7WExGBXkj57jfkFVbgL5N6I8DXw95NIyIiO+AAeCIiJyUSiZAwKAwvTumL0nIdlv77JH75zXLJZSIi6viYFBARObkeEb74v6cGQOnjhvc/T8eXP2bDwJGlREROhUkBERHB39sdr8yIw9Begfjvj9n41xdnUVFVbe9mERFRG2FSQEREAACpxAXPjI3BtIe74ezlQryx6SSu3yy/+4lEROTwmBQQEZFAJBLh4QGheOmPfVFZpcebm07i1EW1vZtFREStjEkBERFZiArrhNdm3Y/O/nJ8kHoOXxzOssvSyURE1DaYFBARkVWdPGVYOK0/HowNxlfHrmJVShrKKvX2bhYREbUCJgVERNQoiasYTz0ajScTonDh2i0s3fgzrhVwkyIioo6GSQEREd3VQ307Y+G0/qiuMeDtzafwU0a+vZtEREQtiEkBERHdk66dvfHaUwMRHuiJtV9mYOu3v6LGYLB3s4iIqAUwKSAionvmrZDh5an9EN8/BPt+zsF7W39BaYXO3s0iIqJmYlJAREQ2cXURY/qY7nhmbAyy8kqxdOPPuJJfau9mERFRMzApICKiJhnWOwiLZ/SHCMDbm0/jx/Qb9m4SERE1EZMCIiJqsohAL7z61EB0C/HGhq8zsXnfRVTXcJ4BEZGjYVJARETN4uUhxQtTYvHI/aE4dPo6lm85g5Iyrb2bRURENmBSQEREzeYiFmPKqG6Y+1hPXCu4jb9v/Bm/XS+xd7OIiOgeMSkgIqIWM6hHAJbMHACpqxj/+PQ0vjtzHUaj0d7NIiKiu2BSQERELSpUpcCrTw5ETEQnbNp7ERu/uQB9dY29m0VERHfApICIiFqcwl2C5yfHYtzQcPyQfgPvfHoGRaVV9m4WERE1gkkBERG1CrFYhEkPdsWfJ/ZGXmE5lm78GRev3bJ3s4iIyAomBURE1KriopR4NWkAPNwkWLH1F+w/mcN5BkRE7QyTAiIianXB/nL8LWkAenfxw5YDv2L97gxo9ZxnQETUXjApICKiNuHh5or5/683Hn8gEj+dL8Cy/5zCzeJKezeLiIjApICIiNqQWCTCY8Mi8T+T+0BTXIWl/z6J81eK7N0sIiKn52rPi+t0OqxevRo7d+5EaWkpoqOjsWDBAgwZMuSO56Wnp2PHjh1IT0/HpUuXoNfrcfHiRYt6OTk5+Oc//4lz585Bo9FAJpPhvvvuw5w5czBixAizulFRUY1eb+jQofjkk08AALm5uYiPj7dab926dXjwwQfv8qmJiCj2Pn/835MD8K8dZ/HPbb9g8kNdkTAoDCKRyN5NIyJySnZNChYtWoR9+/YhKSkJ4eHhSE1NxZw5c7B582b069ev0fMOHz6MlJQUREVFITQ0FJcvX7Zar6CgAMXFxRg/fjwCAwNRVVWF/fv3Y+7cuXjrrbcwefJkoe7y5cstzj937hw2bdqEYcOGWRx77LHHMHz4cLOy6Ojoe/3oREROL8DXA0uS4rDh6wtI+S4LV/JvY1ZiNNykdv2niYjIKdntmzc9PR1fffUVFi9ejKeeegoA8Pjjj2PcuHFYsWIFPv3000bPnTp1KubMmQM3Nze89dZbjSYFAwYMEH7hN5kxYwYmTZqEjRs3miUFEyZMsDj/xIkTEIlEGDdunMWxnj17Wj2HiIjunZvUFc9O6Ik9QZ74/Lss5BWWY/6k3gjo5GHvphERORW7zSnYs2cPJBIJnnjiCaFMJpNh8uTJOHXqFNRqdaPn+vv7w83NrUnXFYvFCAwMRGlp6R3r6XQ67Nu3DwMHDkRgYKDVOhUVFdDpdE1qBxER1RKJRHh0UDhe+ENfFN/W4o2NJ5GeddPezSIicip2SwoyMzMRGRkJuVxuVt6nTx8YjUZkZma22LUqKytRVFSEa9euYePGjfj+++/vOm/h8OHDKC0txWOPPWb1+OrVq9GvXz/06dMHU6ZMwc8//9xi7SUickY9I33xf08NhL+3G1anpOPLI9kwcD8DIqI2YbfhQxqNBgEBARblSqUSAO74pMBW77//PjZs2ACg9knBmDFjsGTJkjues2vXLkilUjzyyCNm5WKxGMOHD8fo0aOhUqlw9epVJCcnY9asWdi4cSMGDBjQYu0mInI2Sh93LJ4Zh017LuC/P2Tjav5tzB7XA+4yzjMgImpNdvuWraqqgkQisSiXyWQAAK1W22LXmjJlCh544AGo1Wrs3bsXNTU1dxz2U1ZWhu+++w4PPfQQvLy8zI4FBwcjOTnZrCwxMRFjx47FihUrsHXrVpvb5+ensPmclqJUetrt2o6I8bIN42Ubxqve4lmDsOvHy0j+8jze/s9pLJl1P0IDzOPDeNmG8bIN42Ubxss27TFedksK3NzcoNfrLcpNyYApOWgJERERiIiIAFA7mXnOnDmYN28eUlJSrC5/t3fvXmi1WowfP/6e3j8gIABjx47F9u3bUVlZCXd3d5vaV1hYBoOh7R+RK5We0Ghut/l1HRXjZRvGyzaMl6Uh0Sr4ekjw4X/PYcGqw5g9tgfiomqfJjNetmG8bMN42Ybxso294iUWi+74Q7TdkgKlUml1iJBGowEAqFSqVrv2I488giVLliA7OxtdunSxOL5r1y54enpi5MiR9/yeQUFBMBgMKC0ttTkpICIi66LCOuG1pwbig9Sz+CD1LPp288O1gjLcKtXC10uGSQ91xZCe1heDICKie2e3icbR0dHIzs5GeXm5WXlaWppwvLWYnkaUlZVZHFOr1Th+/DjGjBkDqVR6z++Zk5MDFxcXeHt7t1g7iYgI8PVyw6Lp/REV6o1ffi1EUakWRgCFpVr8+5sLOHY+395NJCJyeHZLChISEqDX65GSkiKU6XQ67NixA/379xcmIefl5SErK6tJ1ygqKrIoq66uRmpqKmQyGbp27Wpx/Ouvv4bBYGh06JC197x69Sq++uorDBgwoMlLpRIRUeMkri64WVJlUa6rNuDT/Zdw8dotlFdZDkklIqJ7Y7fhQ7GxsUhISMCKFSug0WgQFhaG1NRU5OXlYdmyZUK9hQsX4sSJE7h48aJQdv36dezcuRMAcPbsWQDAhx9+CKD2CcOoUaMAAO+++y6uXr2KwYMHIygoCDdv3sSuXbuQlZWFl19+2WI5VAD48ssvoVKpMGjQIKvtfvfdd5GTk4PBgwdDpVLh2rVrwuTihQsXtkBkiIjImsJS6wtQVFRV4x+fnQEA+HrJEKJUIFSlEP4b4OsOF7HdfgMjInIIdl3jbfny5Vi1ahV27tyJkpISREVFYe3atYiLi7vjebm5uVi9erVZmen1xIkThaQgPj4eW7Zswfbt21FcXAx3d3f06NEDCxYswOjRoy3e9/Llyzh//jxmzZoFcSP/gAwbNgxbt27Ff/7zH9y+fRteXl4YNmwY5s+fj27dujUlDEREdA/8vGRWE4NOChmefDQauZoy5KrLkKMpw/nsItTULeDg6iJGZ385QlRyhCoVCFHV/vHyuPchokREHZ3IaOTOMPbG1YccA+NlG8bLNozX3R07n49/f3MBumqDUCZ1FePJR6MtJhvrqw24UVhelyiUI6cuYSgpr1+O2lsuRYhKUZcoyBGiVCDITw6Ja8d7qsD+ZRvGyzaMl224+hAREVEzmG78dxzOQtFdVh+SuIoRFuCJsN/tbVBarhMSBNNThQOnclBdU/vDjItYhEA/j/onCnVDkHwUUqtLWBMRdRRMCoiIyGEM6RmIIT0Dm/xLm5dcip5yX/SM8BXKagwG5BdV1iYKmjLkqMtwKbcYP2UUCHXkbq7CPIUQVW2iEOwvh0zi0iKfi4jI3pgUEBGRU3MR18456OwvxyAECOXlVfq6RKEcOXUJw/fpedDpa4cviURAQCePuiFIcmEokp+3G58qEJHDYVJARERkhdxNgqiwTogK6ySUGYxGaIprnyrk1CUM1/Jv4+SF+s043aQuDeYq1P63s1IOdxn/ySWi9ovfUERERPdILBIhoJMHAjp5IC5KJZRXaquRd7N+QnOOugw/ZeSj8kyNUMff281sqdQQlQIqH3eIxXyqQET2x6SAiIiomdxlruja2RtdO9fvam80GlFYWmW2+lGupgy//HYTpnX/pK5idFbK6+cq1P1X4S6x0ychImfFpICIiKgViEQi+Hu7w9/bHX27+QvlOn0N8grr5imoa5dNPfPrTfyQfkOo08mzwSZsdfsrBPh6wNWl4y2XSkTtA5MCIiKiNiSVuCAi0AsRgV5CmdFoREm5TlgmtXYIUjkyrjTchE2EYD+52VKpISoFvOXchI2Imo9JARERkZ2JRCL4KGTwUcjQq4ufUF5dY0B+YUV9oqApQ8aVIhw9ly/U8fKQmCcKSgWC/T0gceVyqUR075gUEBERtVOuLuLaG36VAuhZX367QodcTbmQKOSoy3DozHXo63Z7FotqN2ELUcoRqlKg531KeMlc0MlTxuVSicgqJgVEREQOxtNDiphwKWLC65dLrTEYoL5VKeypkKsuR9b1UpzIVOOLw5cBAB4yV2HzNdNThc7+csikfKpA5OyYFBAREXUALmIxgvzkCPKT4/6Y+k3YKqqqUaY34NyvauHJwo9nb0Crq10uVQRA1cndbPWjEJUC/t5uEPOpApHTYFJARETUgXm4uSI81BMqz/oJyQajETdLqmqXSW0wufn0RQ3qVkuFTOpSO/zIlCgoa/94uPHWgagj4t9sIiIiJyMWiaDycYfKxx39uyuFcq2uBtdv1i6TmlOXMJzIVOO7X/KEOn5ebsJSqabJzQGdPLgJG5GDY1JAREREAGqfDnQJ9kKXYPPlUm/d1gqJQu2chXKkZxXCULcLm8RVjGD/+qcKtfMV5PD04HKpRI6CSQERERE1SiQSwdfLDb5ebujTtX4TNn11DfJuVtQ/VdCUIS3rJn48W78Jm49CajZXIVSpQKAfN2Ejao+YFBAREZHNJK4uCA/0RHigp1m5sAmbsApSGfZfzUF1Te1TBRexCEF+coSq5GYJg7dcyuVSieyISQERERG1GG+5FN6RvugZ6SuUVdcYUFBk2oStds7ChWvFOHa+QKijcJcIy6SGqGr3Vwj2k0Mq4XKpRG2BSQERERG1KlcXMTorFeisVAA96svLKvW43mD4UY66HId/uQ5d3SZsIhEQ6OtRlyiYnirI4eflxqcKRC2MSQERERHZhcJdgqiwTogKq9+EzWAwQlNc2SBRKEP2jVL8fEEt1HGXuSJUKRf2VAhVKtBZKYeblLc1RE3Fvz1ERETUbojFIgT4eiDA1wMDolVCeaW2Gtc15cKeCjmaMhw9l4+quk3YAEDl4163p4K8btlUBZQ+7tyEjegeMCkgIiKids9d5or7QrxxX4i3UGY0GlFYUtUgUShHrroMZ37VoG61VEglYmHjNdNSqSEqBeRuEjt9EqL2iUkBEREROSSRSAR/H3f4+7ijX7cGm7Dpa5B3s9xst+ZTF9X4Pq1+EzZfL1mDRKH2qUKgrztcxFwulZwTkwIiIiLqUGQSF0QGeSEyyHwTtuIynbBMqilZOJ9dhBpD7WMFVxcxOvvLcV+YD5SeMmHOghc3YSMnYNekQKfTYfXq1di5cydKS0sRHR2NBQsWYMiQIXc8Lz09HTt27EB6ejouXboEvV6PixcvWtTLycnBP//5T5w7dw4ajQYymQz33Xcf5syZgxEjRpjVXbRoEVJTUy3eIzY2Ftu3bzcrMxgMSE5OxpYtW6DRaBAREYFnn30WiYmJtgeBiIiIWp1IJEInTxk6ecrQu4ufUF5dY8CNwgrkqG8jV107Z+H0BTVu3dYKdbzlUrPVj0KUCgT5ySFx5VMF6jjsmhQsWrQI+/btQ1JSEsLDw5Gamoo5c+Zg8+bN6NevX6PnHT58GCkpKYiKikJoaCguX75stV5BQQGKi4sxfvx4BAYGoqqqCvv378fcuXPx1ltvYfLkyWb13d3d8frrr5uV+fr64vdWrlyJtWvXYsqUKejVqxe+/fZbLFiwAGKxGAkJCU2IBBEREdmDq4sYoaraYUQmSqUnsq4U/u6pQjkOnMpFdU3tcqkuYhEC/TyEzddMQ5F8FNyEjRyTyGg0TcVpW+np6XjiiSewePFiPPXUUwAArVaLcePGQaVS4dNPP2303Js3b0KhUMDNzQ1vvfUWNm3aZPVJgTUGgwGTJk1CdXU1du/eLZQvWrQIBw4cwMmTJ+94fkFBAeLj4zF16lQsWbIEQO0jyRkzZuDGjRs4cOAAxDaORywsLIPB0Pb/NyiVntBobrf5dR0V42Ubxss2jJdtGC/bMF62aSxeNQYDCooqhaVSc+uWTS0srX+qIHdzNZunEKpSINhfDlkH3oSN/cs29oqXWCyCn5+i0eN2e1KwZ88eSCQSPPHEE0KZTCbD5MmTsXLlSqjVaqhUKqvn+vv7N/m6YrEYgYGByMjIsHq8pqYGlZWVUCisB+3AgQPQ6/WYNm2aUCYSiTB16lS8+OKLSE9PR9++fZvcPiIiImqfXMRiBPvLEewvx/0xAUJ5RZUeuZpyYW+FXHUZfki/Aa2+drlUEQCVrwdCGyyVGqpUwM+bm7BR+2G3pCAzMxORkZGQy+Vm5X369IHRaERmZmajSYGtKisrUVlZibKyMhw8eBDff/89xo8fb1GvvLwccXFxqKyshI+PDx5//HG88MILkMlkZu1WKBSIjIy0aDcAZGRkMCkgIiJyIh5uEnQP9UH3UB+hzGA04mZxJXLU5UKicE1dhpMXNUIdN6lLg7kK9Zuwucu4Dgy1Pbv1Oo1Gg4CAAItypbJ2STG1Wm1xrKnef/99bNiwAUDtk4IxY8YIQ38aXnf27NmIiYmBwWDAoUOHsHHjRmRlZWH9+vVm7bb2pKI12k1ERESOSSwSQdXJA6pOHoiLql8utUpXjes3y+uHH6nL8FNGASrPXBfq+Hu7CUOQTE8WVD7uEIv5VIFaj92SgqqqKkgklhuHmH6V12q1FseaasqUKXjggQegVquxd+9e1NTUQKfTmdV58cUXzV6PGzcOAQEBSE5OxpEjRzBs2DCh3VKp5dJkzWn3ncZ3tTal0tNu13ZEjJdtGC/bMF62Ybxsw3jZpjXjFdq5EwY3eG00GqEprsSVG6W4klda+98bJUj77SYMwiZsLggP9EREkBcigr0QGeSNiGAveLaT5VLZv2zTHuNlt6TAzc0Ner3eotx0U91wyE5zRUREICIiAgDw+OOPY86cOZg3bx5SUlLuOJbv6aefRnJyMo4dOyYkBW5ubhYJRXPbzYnGjoHxsg3jZRvGyzaMl20YL9vYI14iAJFKOSKVciA2CACg09fULZdaJkxuPnb2BvafuCac18lTVjepuXa+QqhSgQBfD7i6tN1yqexftuFE499RKpVWh9poNLVj7VpqPoE1jzzyCJYsWYLs7Gx06dKl0Xr+/v6QSCQoKSkRypRKpdUVitqi3UREROQ8TE8HwgPrf1U2Go0oLdcJy6SaEoaMKw03YRMh2E9utlRqiEoBb3n7eKpA7ZPdkoLo6Ghs3rwZ5eXlZpON09LShOOtxfSrfllZ2R3r5efnQ6/Xm+1VEBMTg5SUFGRnZ5tNNja1OyYmphVaTERERFS74qG3QgZvhQy9Is03YcsvqjDbVyHjShGOnssX6nh5SMwTBaUCwf4ekLh23OVS6d7ZLSlISEjAhg0bkJKSIuxToNPpsGPHDvTv31+YhJyXl4fKykp07drV5msUFRVZbD5WXV2N1NRUyGQy4T21Wi30er3FMqQffvghAGD48OFCWXx8PJYtW4bPPvvMbJ+CrVu3Ijg4GLGxsTa3k4iIiKg5XF3EtcOIlAqz+Qq3K3TI1ZQ3SBbKcOjMdeirazdhE4tqN2ELMS2XWpcwdPKUcblUJ2O3pCA2NhYJCQlYsWIFNBoNwsLCkJqairy8PCxbtkyot3DhQpw4ccJsc7Lr169j586dAICzZ88CqL+Bj46OxqhRowAA7777Lq5evYrBgwcjKCgIN2/exK5du5CVlYWXX35ZeEKh0WgwceJEjBs3Dl26dBFWHzp27BgSExMxcOBA4dqBgYFISkrChg0boNVq0bt3b2HTs5UrV9q8cRkRERFRa/H0kCImXIqY8E5CmcFgRMGtivq9FdRluJxXihOZ9cO6PWSuDZZLlSNU5YnO/nLIpHyq0FHZdSHc5cuXY9WqVdi5cydKSkoQFRWFtWvXIi4u7o7n5ebmYvXq1WZlptcTJ04UkoL4+Hhs2bIF27dvR3FxMdzd3dGjRw8sWLAAo0ePFs718vLCiBEjcOTIEaSmpsJgMCAiIgKLFi1CUlKSxfVfeukleHt7Y9u2bdixYwciIyPx3nvvITExsbkhISIiImpVYrEIQX5yBPnJMTC6fi5kRVU1rt8sq3uqUPt04cdzN6DVNdiErZO72d4KISqFXVdRpJYjMhqNbb/sDZnh6kOOgfGyDeNlG8bLNoyXbRgv2zBe9QxGIwpLqsyGH+VoyqEuqoDpzsVd5oJgf3l9olA3jMnDjZuwWcPVh4iIiIjIoYhFIih93KH0cUe/7vWbsGl1NcgrrB1+VFimw69Xi/DzBTW++yVPqOPnVbcJm0ouzFUI6OTBTdjaKSYFRERERGQTmdQFkUFeiAzyEn75NhqNuHVbK+ypYJrgnJ5VCEPdwBSJq9jsqUKosnbp1PayCZszY1JARERERM0mEong6+UGXy839OnqL5Trqw24UVi/p0Kuugzplwvx49kbQh1vhVTYfM00ZyHQr203YXN2TAqIiIiIqNVIXMVH45aqAAAYu0lEQVQIC/BEWICnWXlJuU5IEkxzFvZfzUF1Te1TBZe6CdGhKrnZ5GZvuZTLpbYCJgVERERE1Oa85VJ4y33RM6J+T6nqGgMKblXWJgp1w5Au5hTj2PkCoY7CXSLsqVC7XKoCwX5ySCVcLrU5mBQQERERUbvg6iJGZ385OvvLMQgBQnl5lb4uUagdhpSjLsPhtOvQ6Ws3YROJgEBfj7pEoX5/BT8vNz5VuEdMCoiIiIioXZO7SRAV1glRYeabsGmKK4W5CjnqMlzJL8XPF+o3YXOXuQqTmU3JQmelHG5S3gL/HiNCRERERA5HLBYhwNcDAb4eGNBgE7ZKbTWu3yw321vh2Pl8VJ6uEeqofNzr9lSQ1y2bqoDSxx1iJ36qwKSAiIiIiDoMd5kr7uvsjfs6ewtlRqMRhaVVyFWXC4lCrqYMZ37VwLSNr1QiFjZeC61LGEJUCsjdJHb6JG2LSQERERERdWgikQj+3u7w93ZH3271y6Xq9PWbsOWqy5GrKcPpSxp8n1a/CZuvl6xBolD7VCHQ1x0u4o61XCqTAiIiIiJySlKJCyICvRAR6CWUGY3G2uVSGww/ylGX43x2EWoMtY8VXF3ECPb3qN9XoW4IktcdNmE7dj4fOw5noahUC18vGSY91BVDega2+me8V0wKiIiIiIjqiEQi+Chk8FHI0KuLn1BeXWPAjcIKs2Th3JUiHDmXL9TxlkvNVj8KUSoQ5CfHyYtq/PubC9BV166WVFiqxb+/uQAA7SYxYFJARERERHQXri7i2l2XVQoMaVBeWqHDdXUZcjT1k5u/PZ0LfV0C4CKunbxsespgoqs2YMfhLCYFRERERESOzstDCq8IX8Q02IStxmCA+lb9cqm7j161em5hqbatmnlXHWuGBBERERGRnbmIxQjyk+P+mABMerAr/LxkVus1Vm4PTAqIiIiIiFrRpIe6QupqftstdRVj0kNd7dQiSxw+RERERETUikzzBrj6EBERERGRExvSMxBDegZCqfSERnPb3s2xwOFDREREREROjkkBEREREZGTY1JAREREROTkmBQQERERETk5JgVERERERE6OSQERERERkZNjUkBERERE5OSYFBAREREROTkmBURERERETo47GrcDYrHIKa/tiBgv2zBetmG8bMN42Ybxsg3jZRvGyzb2iNfdrikyGo3GNmoLERERERG1Qxw+RERERETk5JgUEBERERE5OSYFREREREROjkkBEREREZGTY1JAREREROTkmBQQERERETk5JgVERERERE6OSQERERERkZNjUkBERERE5OSYFBAREREROTlXezeAmk6n02H16tXYuXMnSktLER0djQULFmDIkCF3PbegoABvv/02jhw5AoPBgMGDB2Px4sUIDQ21qJuSkoINGzYgNzcXwcHBSEpKwvTp01vjI7WqpsZr3759+Prrr5Geno7CwkIEBQVh5MiReO655+Dp6WlWNyoqyup7/P3vf8fUqVNb7LO0habGa82aNfjXv/5lUe7v748jR45YlDt7/xo1ahSuX79u9Vh4eDj27dsnvO5I/UutVmPTpk1IS0vDuXPnUFFRgU2bNmHQoEH3dH5WVhbefvttnD59GhKJBCNHjsTChQvh6+trVs9gMCA5ORlbtmyBRqNBREQEnn32WSQmJrbGx2o1TY2XwWBAamoq9u/fj8zMTJSUlCAkJATjxo3D008/DalUKtTNzc1FfHy81fdZt24dHnzwwRb9TK2pOf1r0aJFSE1NtSiPjY3F9u3bzcqcvX8BjX8vAcDQoUPxySefAOg4/Ss9PR2pqak4fvw48vLy4OPjg379+uH5559HeHj4Xc9vz/dfTAoc2KJFi7Bv3z4kJSUhPDwcqampmDNnDjZv3ox+/fo1el55eTmSkpJQXl6OefPmwdXVFRs3bkRSUhL++9//wtvbW6i7detWvPbaa0hISMCsWbNw8uRJLF26FFqtFk8//XRbfMwW09R4vfrqq1CpVJgwYQKCg4Nx8eJFbN68GT/88AO++OILyGQys/rDhw/HY489ZlYWGxvbKp+pNTU1XiZLly6Fm5ub8Lrh/zZh/wJeeeUVlJeXm5Xl5eVh1apVGDZsmEX9jtK/srOzsW7dOoSHhyMqKgpnzpy553Pz8/Mxffp0eHl5YcGCBaioqMCGDRtw6dIlbN++HRKJRKi7cuVKrF27FlOmTEGvXr3w7bffYsGCBRCLxUhISGiNj9YqmhqvyspKvPLKK+jbty/++Mc/ws/PD2fOnMHq1avx008/YePGjRbnPPbYYxg+fLhZWXR0dEt8jDbTnP4FAO7u7nj99dfNyn6fcALsXwCwfPlyi7Jz585h06ZNVr/DHL1/rV+/HqdPn0ZCQgKioqKg0Wjw6aef4vHHH8fnn3+Orl27Nnpuu7//MpJDSktLM3bv3t34ySefCGVVVVXGhx9+2Dht2rQ7nrt27VpjVFSU8fz580LZb7/9ZoyJiTGuWrVKKKusrDTef//9xmeffdbs/BdffNHYr18/Y2lpact8mDbQnHj99NNPFmWpqanG7t27G7/44guz8u7duxvffPPNFmmzPTUnXu+//76xe/fuxpKSkjvWY/9q3AcffGDs3r278dSpU2blHaV/GY1G4+3bt41FRUVGo9Fo3L9/v7F79+5W/65Z89prrxn79u1rzM/PF8qOHDli7N69uzElJUUoy8/PN/bs2dMsZgaDwTht2jTjyJEjjTU1NS30aVpfU+Ol1Wot+pHRaDSuWbPG4j1ycnIs+rGjak7/WrhwoTEuLu6u9di/GvfKK68Yo6KijDdu3BDKOkr/OnXqlFGr1ZqVZWdnG3v16mVcuHDhHc9t7/dfnFPgoPbs2QOJRIInnnhCKJPJZJg8eTJOnToFtVrd6Ll79+5F37590aNHD6Gsa9euGDJkCL755huh7Pjx4yguLsa0adPMzp8+fTrKy8vx/ffft+Anal3NiZe1x6cPP/wwgNohDNZUVVVBq9U2s9X205x4mRiNRpSVlcFoNFo9zv7VuN27dyMkJAT9+/e3etzR+xcAKBQKdOrUqUnn7tu3D6NGjUJAQIBQNnToUERERJh9hx04cAB6vd6sj4lEIkydOhXXr19Henp60z9AG2tqvKRSqdV+NHr0aACNf4dVVFRAp9PZfL32ojn9y6SmpgZlZWWNHmf/sk6n02Hfvn0YOHAgAgMDrdZx5P7Vv39/s2F3ABAREYFu3bo1+vfJpL3ffzEpcFCZmZmIjIyEXC43K+/Tpw+MRiMyMzOtnmcwGHDx4kX06tXL4ljv3r1x5coVVFZWAgAyMjIAwKJuz549IRaLheOOoKnxaszNmzcBwOqX6Oeff46+ffuiT58+GD9+PPbv39/0httJS8RrxIgRiIuLQ1xcHBYvXozi4mKz4+xf1mVkZCArKwvjxo2zerwj9K/mKCgoQGFhodXvsD59+pjFOjMzEwqFApGRkRb1ADhUH2tpd/oOW716Nfr164c+ffpgypQp+Pnnn9u6eXZXXl4ufH8NGjQIy5Yts0jE2b+sO3z4MEpLSy2GOZp0xP5lNBpx8+bNOyZWjnD/xTkFDkqj0Zj9SmaiVCoBoNFfJouLi6HT6YR6vz/XaDRCo9EgLCwMGo0GUqkUPj4+ZvVMZbb++mlPTY1XY9atWwcXFxeMGTPGrLxfv35ITExESEgIbty4gU2bNmH+/Pl47733Gr3Ja4+aEy8vLy/MnDkTsbGxkEgk+Omnn7Bt2zZkZGQgJSVF+IWF/cu6Xbt2AYDVf1A7Sv9qDlMsG/sOKywsRE1NDVxcXKDRaODv72+1XsP3ckbr16+Hp6en2dhusViM4cOHY/To0VCpVLh69SqSk5Mxa9YsbNy4EQMGDLBji9uOUqnE7NmzERMTA4PBgEOHDmHjxo3IysrC+vXrhXrsX9bt2rULUqkUjzzyiFl5R+5fX375JQoKCrBgwYJG6zjC/ReTAgdVVVVlNpnOxDTptbGhBaby3z/6anhuVVXVHa9hqutIwxeaGi9rdu3ahc8//xxz585FWFiY2bGtW7eavZ44cSLGjRuHd999F2PHjoVIJGpC69tec+L15JNPmr1OSEhAt27dsHTpUvz3v//FH/7whztew3QdZ+xfBoMBX331FXr06GF1slpH6V/Nca/fYXK5HFVVVXes50h9rCV99NFHOHr0KJYuXWq2glpwcDCSk5PN6iYmJmLs2LFYsWKFRf/rqF588UWz1+PGjUNAQACSk5Nx5MgRYfIs+5elsrIyfPfdd3jooYfg5eVldqyj9q+srCwsXboUcXFxmDBhQqP1HOH+i8OHHJSbmxv0er1Fuamj/H5FHBNTubWxfKZzTavEuLm5NTrmT6vVNnqN9qip8fq9kydPYsmSJRgxYgT++te/3rW+h4cH/vjHPyI/Px+XL1+2rdF21FLxMpk6dSrc3d1x7Ngxs2uwf5k7ceIECgoKMH78+Huq76j9qzla4jusqf24I/j666+xatUqTJkyBVOmTLlr/YCAAIwdOxZpaWnC0AZnZFrt5V6+w5y5f+3duxdarfaev8McvX9pNBrMnTsX3t7eWL16NcTixm+rHeH+i0mBg1IqlVYfH2k0GgCASqWyep6Pjw+kUqlQ7/fnikQi4dGWUqmEXq+3GAuu0+lQXFzc6DXao6bGq6ELFy7g2WefRVRUFFauXAkXF5d7unZQUBAAoKSkxIYW21dLxKshsViMgIAAsxiwf1natWsXxGIxxo4de8/XdsT+1RymWDb2Hebn5yf83VQqlcLY+d/Xa/hezuLIkSP43//9X4wcORKvvfbaPZ8XFBQEg8GA0tLSVmxd++bv7w+JRGLxHcb+ZW7Xrl3w9PTEyJEj7/kcR+1ft2/fxpw5c3D79m2sX7/e6rCghhzh/otJgYOKjo5Gdna2xfrmaWlpwnFrxGIxunfvjnPnzlkcS09PR3h4ONzd3QEAMTExAGBR99y5czAYDMJxR9DUeJlcu3YNs2fPhq+vLz7++GN4eHjc87VzcnIAWF/jur1qbrx+T6/X48aNG2aTsNi/zJlW7Lj//vutzk9ojCP2r+YICAiAr69vo99hDftNTEwMysrKkJ2dbVbP9P+LI/Wx5kpLS8P8+fPRu3dvm37UAGr7mIuLi9ka6s4mPz8fer3e7O8Z+5c5tVqN48ePY8yYMVaHyDTGEfuXVqvFvHnzcOXKFXz88cfo0qXLXc9xhPsvJgUOKiEhAXq9HikpKUKZTqfDjh070L9/f+GmIi8vz2KJrEceeQS//PKL2ez1y5cv46effjLbbGXw4MHw8fHBZ599Znb+li1b4OHh4TC7DwLNi5dGo8HTTz8NkUiE5OTkRm++ioqKLMpu3bqFzz77DCEhIYiIiGi5D9TKmhMva3FITk6GVqvFAw88IJSxf5kzrdjR2GP3jtS/bHHt2jVcu3bNrGzMmDE4ePAgCgoKhLJjx47hypUrZt9h8fHxkEgkZn3MaDRi69atCA4OdshN3+7GWryysrLwpz/9CZ07d8ZHH31kdSNBwHofu3r1Kr766isMGDCg0fMc2e/jpdVqrS5D+uGHHwKA2cRs9i9zX3/9NQwGg03fYY7Yv2pqavD888/jl19+werVq9G3b1+r9Rzx/osTjR1UbGwsEhISsGLFCmG2empqKvLy8rBs2TKh3sKFC3HixAlcvHhRKJs2bRpSUlLwpz/9CbNmzYKLiws2btwIpVKJp556Sqjn5uaG//mf/8HSpUvx17/+FcOHD8fJkyfx5Zdf4qWXXrKYRNSeNSdes2fPRk5ODmbPno1Tp07h1KlTwrGwsDBht9pPP/0U3377LUaMGIHg4GAUFBRg27ZtKCoqwgcffNB2H7YFNCdeI0eORGJiIrp37w6pVIrjx49j7969iIuLM1shh/3LXGMrdph0pP5lYrrRMv3DuXPnTpw6dQpeXl6YMWMGAAjfSQcPHhTOmzdvHvbs2YOkpCTMmDEDFRUVSE5ORnR0tNlEv8DAQCQlJWHDhg3QarXo3bs3Dhw4gJMnT2LlypV3HP/bHjUlXmVlZXjmmWdQWlqKZ555Bt99953Ze0ZFRQlPst59913k5ORg8ODBUKlUuHbtmjD5c+HCha398VpcU+Kl0WiECfxdunQRVh86duwYEhMTMXDgQOH92b/Mffnll1CpVFb39gE6Tv965513cPDgQYwcORLFxcXYuXOncEwulwv7GDni/ZfI2NjOQtTuabVarFq1Crt27UJJSQmioqLwwgsvYOjQoUKdmTNnWr0Jyc/Px9tvv40jR47AYDBg0KBBWLJkCUJDQy2us337dmzYsAG5ubkICgrCzJkzkZSU1Oqfr6U1NV5RUVGNvufEiRPxzjvvAAB+/PFHJCcn49KlSygpKYGHhwf69u2LuXPnIi4urvU+WCtparz+9re/4fTp07hx4wb0ej06d+6MxMREzJ071+ovQc7ev4DaG7ehQ4fioYcewpo1a6y+f0frX0Djf7c6d+4s3HSMGjUKgOVNyK+//op33nkHp06dgkQiwYgRI7B48WKLJ3kGgwHr1q3Dtm3boFarERkZiblz5zrkEq5NiVdubi7i4+Mbfc/58+fjL3/5C4DaTfO2bt2K3377Dbdv34aXlxfuv/9+zJ8/H926dWvJj9ImmhKv0tJSvPHGG0hLS4NarYbBYEBERAQmTpyIpKQki2FXzt6/TC5fvoxHH30Us2bNwqJFi6y+T0fpX6bvcWsaxsoR77+YFBAREREROTnHerZFREREREQtjkkBEREREZGTY1JAREREROTkmBQQERERETk5JgVERERERE6OSQERERERkZNjUkBERERE5OSYFBARkVOaOXOmsBkTEZGzc7V3A4iIqOM4fvz4HXfcdHFxQUZGRhu2iIiI7gWTAiIianHjxo3Dgw8+aFEuFvMBNRFRe8SkgIiIWlyPHj0wYcIEezeDiIjuEX+yISKiNpebm4uoqCisWbMGu3fvxvjx49G7d2+MGDECa9asQXV1tcU5Fy5cwJ///GcMGjQIvXv3RmJiItatW4eamhqLuhqNBm+++Sbi4+PRq1cvDBkyBLNmzcKRI0cs6hYUFOCFF17AwIEDERsbi2eeeQbZ2dmt8rmJiNorPikgIqIWV1lZiaKiIotyqVQKhUIhvD548CBycnIwffp0+Pv74+DBg/jXv/6FvLw8LFu2TKh39uxZzJw5E66urkLdQ4cOYcWKFbhw4QLee+89oW5ubi6mTp2KwsJCTJgwAb169UJlZSXS0tJw9OhRDBs2TKhbUVGBGTNmIDY2FgsWLEBubi42bdqE5557Drt374aLi0srRYiIqH1hUkBERC1uzZo1WLNmjUX5iBEj8PHHHwuvL1y4gM8//xw9e/YEAMyYMQPz58/Hjh07MGXKFPTt2xcA8NZbb0Gn02Hr1q2Ijo4W6j7//PPYvXs3Jk+ejCFDhgAAXn/9dajVaqxfvx4PPPCA2fUNBoPZ61u3buGZZ57BnDlzhDJfX1+8++67OHr0qMX5REQdFZMCIiJqcVOmTEFCQoJFua+vr9nroUOHCgkBAIhEIsyePRsHDhzA/v370bdvXxQWFuLMmTMYPXq0kBCY6j777LPYs2cP9u/fjyFDhqC4uBg//PADHnjgAas39L+f6CwWiy1WSxo8eDAA4OrVq0wKiMhpMCkgIqIWFx4ejqFDh961XteuXS3K7rvvPgBATk4OgNrhQA3LG+rSpQvEYrFQ99q1azAajejRo8c9tVOlUkEmk5mV+fj4AACKi4vv6T2IiDoCTjQmIiKndac5A0ajsQ1bQkRkX0wKiIjIbrKysizKfvvtNwBAaGgoACAkJMSsvKHLly/DYDAIdcPCwiASiZCZmdlaTSYi6pCYFBARkd0cPXoU58+fF14bjUasX78eAPDwww8DAPz8/NCvXz8cOnQIly5dMqu7du1aAMDo0aMB1A79efDBB/H999/j6NGjFtfjr/9ERNZxTgEREbW4jIwM7Ny50+ox080+AERHR+PJJ5/E9OnToVQq8e233+Lo0aOYMGEC+vXrJ9RbsmQJZs6cienTp2PatGlQKpU4dOgQfvzxR4wbN05YeQgAXn31VWRkZGDOnDl4/PHH0bNnT2i1WqSlpaFz5854+eWXW++DExE5KCYFRETU4nbv3o3du3dbPbZv3z5hLP+oUaMQGRmJjz/+GNnZ2fDz88Nzzz2H5557zuyc3r17Y+vWrXj//fexZcsWVFRUIDQ0FC+99BKefvpps7qhoaH44osv8MEHH+D777/Hzp074eXlhejoaEyZMqV1PjARkYMTGfkslYiI2lhubi7i4+Mxf/58/OUvf7F3c4iInB7nFBAREREROTkmBURERERETo5JARERERGRk+OcAiIiIiIiJ8cnBURERERETo5JARERERGRk2NSQERERETk5JgUEBERERE5OSYFREREREROjkkBEREREZGT+//9MTAzgzQI7gAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 864x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkyubuJSOzg3",
        "colab_type": "text"
      },
      "source": [
        "# 5. Performance On Test Set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tg42jJqqM68F",
        "colab_type": "text"
      },
      "source": [
        "### 5.1. Data Preparation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xWe0_JW21MyV",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "We'll need to apply all of the same steps that we did for the training data to prepare our test data set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hba10sXR7Xi6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "7021bb28-ff79-4356-d42c-9b1fcf169f15"
      },
      "source": [
        "# Prediction on test set\n",
        "\n",
        "print('Predicting labels for {:,} test sentences...'.format(len(prediction_inputs)))\n",
        "\n",
        "# Put model in evaluation mode\n",
        "model.eval()\n",
        "\n",
        "# Tracking variables \n",
        "predictions , true_labels = [], []\n",
        "\n",
        "# Predict \n",
        "for batch in prediction_dataloader:\n",
        "  # Add batch to GPU\n",
        "  batch = tuple(t.to(device) for t in batch)\n",
        "  \n",
        "  # Unpack the inputs from our dataloader\n",
        "  b_input_ids, b_input_mask, b_labels = batch\n",
        "  \n",
        "  # Telling the model not to compute or store gradients, saving memory and \n",
        "  # speeding up prediction\n",
        "  with torch.no_grad():\n",
        "      # Forward pass, calculate logit predictions\n",
        "      outputs = model(b_input_ids, token_type_ids=None, \n",
        "                      attention_mask=b_input_mask)\n",
        "\n",
        "  logits = outputs[0]\n",
        "\n",
        "  # Move logits and labels to CPU\n",
        "  logits = logits.detach().cpu().numpy()\n",
        "  label_ids = b_labels.to('cpu').numpy()\n",
        "  \n",
        "  # Store predictions and true labels\n",
        "  predictions.append(logits)\n",
        "  true_labels.append(label_ids)\n",
        "\n",
        "print('    DONE.')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Predicting labels for 1,374 test sentences...\n",
            "    DONE.\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}