Open_LLM_Training_And_Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.0 Introduction\n",
    "\n",
    "In this notebook we will train and validate 2 Open LLM models to establish a baseline of the accuracy that can be achieved currently with the available Open LLM's.\n",
    "\n",
    "We will compare this baseline against the GPT-3.5 Turbo accuracy and the accuracies achieved by the 3 smaller Transformer models.\n",
    "\n",
    "The open LLM models that will be used are:\n",
    "* PolyLM 1.7B ([HuggingFace URL](https://huggingface.co/DAMO-NLP-MT/polylm-1.7b))\n",
    "* OpenLLaMA 7B V2 ([HuggingFace URL](https://huggingface.co/openlm-research/open_llama_7b_v2))\n",
    "\n",
    "\n",
    "The PolyLM model contains 1.7 billion parameters and the OpenLLaMA model contains 7 billion parameters. The Dutch language was part of the datasets that were used for pre-training these models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Modules\n",
    "import gc\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "from datasets import Dataset, DatasetDict\n",
    "from peft import prepare_model_for_kbit_training, get_peft_model, LoraConfig, TaskType\n",
    "from sklearn.metrics import classification_report, accuracy_score, precision_score, recall_score\n",
    "from transformers import (AutoConfig,\n",
    "                          AutoModelForSequenceClassification, \n",
    "                          AutoTokenizer,\n",
    "                          BitsAndBytesConfig,\n",
    "                          DataCollatorWithPadding, \n",
    "                          TrainingArguments, \n",
    "                          Trainer)\n",
    "\n",
    "# Set Seed for Randomness\n",
    "seed = 33\n",
    "torch.manual_seed(seed)\n",
    "torch.cuda.manual_seed(seed)\n",
    "torch.backends.cudnn.deterministic = False # I prefer some randomness in each training...that gives a good impression of the variations in baseline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0 Load Datasets\n",
    "\n",
    "We will reload the training and validation CSV files that were generated earlier with the notebook 'Prepare_Train_and_Validation_Datasets.ipynb'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3069, 11)\n",
      "(1559, 11)\n"
     ]
    }
   ],
   "source": [
    "# Load Datasets\n",
    "train_df = pd.read_csv('./data/train_df.csv')\n",
    "val_df = pd.read_csv('./data/val_df.csv')\n",
    "\n",
    "# Summary\n",
    "print(train_df.shape)\n",
    "print(val_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>mainSection</th>\n",
       "      <th>published_at</th>\n",
       "      <th>publisher</th>\n",
       "      <th>partisan</th>\n",
       "      <th>url</th>\n",
       "      <th>text_wordcount</th>\n",
       "      <th>max_words_text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10706318</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorlog</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorl...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2018-10-07</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/ogen-als-schoteltjes-bij-de-...</td>\n",
       "      <td>539</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorl...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12633805</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>/amsterdam</td>\n",
       "      <td>2019-05-10</td>\n",
       "      <td>parool</td>\n",
       "      <td>True</td>\n",
       "      <td>www.parool.nl/amsterdam/geen-beeld-maar-een-mo...</td>\n",
       "      <td>662</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7140125</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>/</td>\n",
       "      <td>2017-04-18</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>494</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4490774</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokikker</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokik...</td>\n",
       "      <td>/</td>\n",
       "      <td>2017-03-14</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>291</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokik...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10592180</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2018-09-11</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/meer-fouten-kabinet-bij-steu...</td>\n",
       "      <td>471</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "0  10706318   Ogen als schoteltjes bij de Tachtigjarige Oorlog   \n",
       "1  12633805  Geen beeld, maar een monument voor Mandela in ...   \n",
       "2   7140125  Hoe ga je een onveilige arbeidscultuur zoals i...   \n",
       "3   4490774  Wetenschappers ontdekken lichtgevende discokikker   \n",
       "4  10592180  Meer fouten kabinet bij steun aan strijdgroepe...   \n",
       "\n",
       "                                                text mainSection published_at  \\\n",
       "0  Ogen als schoteltjes bij de Tachtigjarige Oorl...       /home   2018-10-07   \n",
       "1  Geen beeld, maar een monument voor Mandela in ...  /amsterdam   2019-05-10   \n",
       "2  Hoe ga je een onveilige arbeidscultuur zoals i...           /   2017-04-18   \n",
       "3  Wetenschappers ontdekken lichtgevende discokik...           /   2017-03-14   \n",
       "4  Meer fouten kabinet bij steun aan strijdgroepe...       /home   2018-09-11   \n",
       "\n",
       "  publisher  partisan                                                url  \\\n",
       "0     trouw      True  www.trouw.nl/home/ogen-als-schoteltjes-bij-de-...   \n",
       "1    parool      True  www.parool.nl/amsterdam/geen-beeld-maar-een-mo...   \n",
       "2     trouw      True                                                NaN   \n",
       "3     trouw      True                                                NaN   \n",
       "4     trouw      True  www.trouw.nl/home/meer-fouten-kabinet-bij-steu...   \n",
       "\n",
       "   text_wordcount                                     max_words_text  labels  \n",
       "0             539  Ogen als schoteltjes bij de Tachtigjarige Oorl...       1  \n",
       "1             662  Geen beeld, maar een monument voor Mandela in ...       1  \n",
       "2             494  Hoe ga je een onveilige arbeidscultuur zoals i...       1  \n",
       "3             291  Wetenschappers ontdekken lichtgevende discokik...       1  \n",
       "4             471  Meer fouten kabinet bij steun aan strijdgroepe...       1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>mainSection</th>\n",
       "      <th>published_at</th>\n",
       "      <th>publisher</th>\n",
       "      <th>partisan</th>\n",
       "      <th>url</th>\n",
       "      <th>text_wordcount</th>\n",
       "      <th>max_words_text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9266995</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-12-18</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/verdachte-dodelijke-steek...</td>\n",
       "      <td>188</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4130077</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-02-11</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/honderden-arrestaties-bij...</td>\n",
       "      <td>122</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11147268</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2019-01-20</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...</td>\n",
       "      <td>262</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10749100</td>\n",
       "      <td>Klaar voor de verdediging</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-16</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/klaar-voor-de-verdediging...</td>\n",
       "      <td>411</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10700707</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-05</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/windvlaag-grijpt-springma...</td>\n",
       "      <td>286</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "0   9266995  Verdachte dodelijke steekpartijen Maastricht l...   \n",
       "1   4130077  Honderden arrestaties bij acties tegen mensen ...   \n",
       "2  11147268  Waarom de 'oudejaarsbonus' voor de jongeren va...   \n",
       "3  10749100                          Klaar voor de verdediging   \n",
       "4  10700707  Windvlaag grijpt springmatras en doodt 2-jarig...   \n",
       "\n",
       "                                                text mainSection published_at  \\\n",
       "0  Verdachte dodelijke steekpartijen Maastricht l...     /nieuws   2017-12-18   \n",
       "1  Honderden arrestaties bij acties tegen mensen ...     /nieuws   2017-02-11   \n",
       "2  Waarom de 'oudejaarsbonus' voor de jongeren va...       /home   2019-01-20   \n",
       "3  Klaar voor de verdedigingOver ruim een week be...     /nieuws   2018-10-16   \n",
       "4  Windvlaag grijpt springmatras en doodt 2-jarig...     /nieuws   2018-10-05   \n",
       "\n",
       "  publisher  partisan                                                url  \\\n",
       "0        ad     False  www.ad.nl/binnenland/verdachte-dodelijke-steek...   \n",
       "1        ad     False  www.ad.nl/buitenland/honderden-arrestaties-bij...   \n",
       "2     trouw      True  www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...   \n",
       "3        ad     False  www.ad.nl/binnenland/klaar-voor-de-verdediging...   \n",
       "4        ad     False  www.ad.nl/buitenland/windvlaag-grijpt-springma...   \n",
       "\n",
       "   text_wordcount                                     max_words_text  labels  \n",
       "0             188  Verdachte dodelijke steekpartijen Maastricht l...       0  \n",
       "1             122  Honderden arrestaties bij acties tegen mensen ...       0  \n",
       "2             262  Waarom de 'oudejaarsbonus' voor de jongeren va...       1  \n",
       "3             411  Klaar voor de verdedigingOver ruim een week be...       0  \n",
       "4             286  Windvlaag grijpt springmatras en doodt 2-jarig...       0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary\n",
    "val_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 Process Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Proces DataFrame to DataSet function.\n",
    "def process_dataset(tokenizer):\n",
    "    # Tokenize Helper\n",
    "    def preprocess_function(examples):\n",
    "        return tokenizer(examples[\"text\"], \n",
    "                         truncation = True)\n",
    "\n",
    "    # Create DataSets\n",
    "    tdf = pd.DataFrame({\"text\": train_df.max_words_text.values, \"label\": train_df.labels.values})\n",
    "    vdf = pd.DataFrame({\"text\": val_df.max_words_text.values, \"label\": val_df.labels.values})\n",
    "    tds = Dataset.from_pandas(tdf)\n",
    "    vds = Dataset.from_pandas(vdf)\n",
    "\n",
    "    ds = DatasetDict()\n",
    "    ds['train'] = tds\n",
    "    ds['validation'] = vds\n",
    "\n",
    "    # Tokenize Text\n",
    "    ds = ds.map(preprocess_function, batched = True)\n",
    "\n",
    "    # Summary\n",
    "    print(ds)\n",
    "\n",
    "    return ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.0 Evaluation Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(val_preds):\n",
    "    preds, labels = val_preds\n",
    "    preds = np.argmax(preds, axis = 1)\n",
    "    \n",
    "    report = classification_report(labels, preds, digits = 3)\n",
    "    print(report)\n",
    "    \n",
    "    accuracy_val = accuracy_score(labels, preds)\n",
    "    precision_val = precision_score(labels, preds)\n",
    "    recall_val = recall_score(labels, preds)\n",
    "    \n",
    "    return {\"accuracy\": accuracy_val, \"precision\": precision_val, \"recall\": recall_val}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.0 Training and Validation on Subset of the Data\n",
    "\n",
    "In this section we will train and validate the Open LLM model on the loaded and processed CSV files.\n",
    "\n",
    "It is very likely that optimizing the hyperparameters could lead to a further improvement in performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 PolyLM 1.7B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "model_name = 'DAMO-NLP-MT/polylm-1.7b'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3ae9a37b82a8466a85a0a0fddec9556e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/3069 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1c34e305207e41ab9af3b9b59e138c22",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1559 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
      "        num_rows: 3069\n",
      "    })\n",
      "    validation: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
      "        num_rows: 1559\n",
      "    })\n",
      "})\n",
      "LlamaTokenizer(name_or_path='DAMO-NLP-MT/polylm-1.7b', vocab_size=256000, model_max_length=2048, is_fast=False, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'pad_token': '</s>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={\n",
      "\t0: AddedToken(\"<unk>\", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n",
      "\t1: AddedToken(\"<s>\", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n",
      "\t2: AddedToken(\"</s>\", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Create Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name,\n",
    "                                          use_fast = False,\n",
    "                                          legacy = False)\n",
    "tokenizer.pad_token = tokenizer.eos_token\n",
    "\n",
    "# Set Data Collator\n",
    "data_collator = DataCollatorWithPadding(tokenizer = tokenizer, padding = 'longest')\n",
    "\n",
    "# Tokenize dataset\n",
    "ds = process_dataset(tokenizer)\n",
    "\n",
    "# Summary\n",
    "print(tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reduce the memory footprint the LLM model will be quantized to 4-bits and then finetuned with a QLoRA setup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of GPT2ForSequenceClassification were not initialized from the model checkpoint at DAMO-NLP-MT/polylm-1.7b and are newly initialized: ['score.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 12,587,008 || all params: 1,749,676,032 || trainable%: 0.719390776909265\n",
      "PeftModelForSequenceClassification(\n",
      "  (base_model): LoraModel(\n",
      "    (model): GPT2ForSequenceClassification(\n",
      "      (transformer): GPT2Model(\n",
      "        (wte): Embedding(256000, 2048)\n",
      "        (wpe): Embedding(2048, 2048)\n",
      "        (drop): Dropout(p=0.0, inplace=False)\n",
      "        (h): ModuleList(\n",
      "          (0-23): 24 x GPT2Block(\n",
      "            (ln_1): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
      "            (attn): GPT2Attention(\n",
      "              (c_attn): Linear4bit(\n",
      "                (lora_dropout): ModuleDict(\n",
      "                  (default): Dropout(p=0.05, inplace=False)\n",
      "                )\n",
      "                (lora_A): ModuleDict(\n",
      "                  (default): Linear(in_features=2048, out_features=64, bias=False)\n",
      "                )\n",
      "                (lora_B): ModuleDict(\n",
      "                  (default): Linear(in_features=64, out_features=6144, bias=False)\n",
      "                )\n",
      "                (lora_embedding_A): ParameterDict()\n",
      "                (lora_embedding_B): ParameterDict()\n",
      "                (base_layer): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
      "              )\n",
      "              (c_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
      "              (attn_dropout): Dropout(p=0.0, inplace=False)\n",
      "              (resid_dropout): Dropout(p=0.0, inplace=False)\n",
      "            )\n",
      "            (ln_2): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
      "            (mlp): GPT2MLP(\n",
      "              (c_fc): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
      "              (c_proj): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
      "              (act): FastGELUActivation()\n",
      "              (dropout): Dropout(p=0.0, inplace=False)\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (ln_f): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
      "      )\n",
      "      (score): ModulesToSaveWrapper(\n",
      "        (original_module): Linear(in_features=2048, out_features=2, bias=False)\n",
      "        (modules_to_save): ModuleDict(\n",
      "          (default): Linear(in_features=2048, out_features=2, bias=False)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Create Config\n",
    "config = AutoConfig.from_pretrained(model_name,\n",
    "                                    num_labels = 2,                                                            \n",
    "                                    use_cache = False)\n",
    "\n",
    "# Create Quantization Config\n",
    "quantization_config = BitsAndBytesConfig(load_in_4bit = True,\n",
    "                                         bnb_4bit_use_double_quant = True,\n",
    "                                         bnb_4bit_quant_type = 'nf4',\n",
    "                                         bnb_4bit_compute_dtype = torch.bfloat16)\n",
    "\n",
    "# Create Model\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name,\n",
    "                                                           config = config,\n",
    "                                                           device_map = {\"\":0},\n",
    "                                                           quantization_config = quantization_config)\n",
    "\n",
    "# Set Pad Token Id\n",
    "model.config.pad_token_id = tokenizer.pad_token_id\n",
    "\n",
    "# Create LoRA config\n",
    "loraconfig = LoraConfig(r = 64,\n",
    "                        lora_alpha = 16,\n",
    "                        lora_dropout = 0.05,\n",
    "                        bias = 'none',\n",
    "                        task_type = TaskType.SEQ_CLS,\n",
    "                        fan_in_fan_out = True)\n",
    "\n",
    "# Prep for Training\n",
    "model = prepare_model_for_kbit_training(model)\n",
    "\n",
    "# Create QLoRA Model\n",
    "model = get_peft_model(model, loraconfig)\n",
    "model.print_trainable_parameters()\n",
    "\n",
    "# Show Model Summary\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "06f9e2177b8e4da391e756ed484dcb2e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/288 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e5f13f2b2554ad1ba01b2ec4cb7659e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/195 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.936     0.614     0.742       765\n",
      "           1      0.721     0.960     0.823       794\n",
      "\n",
      "    accuracy                          0.790      1559\n",
      "   macro avg      0.829     0.787     0.783      1559\n",
      "weighted avg      0.827     0.790     0.783      1559\n",
      "\n",
      "{'eval_loss': 0.4815058410167694, 'eval_accuracy': 0.7902501603592046, 'eval_precision': 0.7209082308420057, 'eval_recall': 0.9596977329974811, 'eval_runtime': 110.4869, 'eval_samples_per_second': 14.11, 'eval_steps_per_second': 1.765, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ebcef4671354c1ea9ee2dfe776cec8b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/195 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.888     0.775     0.828       765\n",
      "           1      0.807     0.906     0.853       794\n",
      "\n",
      "    accuracy                          0.842      1559\n",
      "   macro avg      0.847     0.840     0.841      1559\n",
      "weighted avg      0.847     0.842     0.841      1559\n",
      "\n",
      "{'eval_loss': 0.3593486249446869, 'eval_accuracy': 0.841565105837075, 'eval_precision': 0.8069584736251403, 'eval_recall': 0.9055415617128464, 'eval_runtime': 110.6219, 'eval_samples_per_second': 14.093, 'eval_steps_per_second': 1.763, 'epoch': 2.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b5224b4607fc4c28a5b9aaf6a16469f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/195 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.842     0.841     0.841       765\n",
      "           1      0.847     0.848     0.847       794\n",
      "\n",
      "    accuracy                          0.844      1559\n",
      "   macro avg      0.844     0.844     0.844      1559\n",
      "weighted avg      0.844     0.844     0.844      1559\n",
      "\n",
      "{'eval_loss': 0.3492984473705292, 'eval_accuracy': 0.8441308531109686, 'eval_precision': 0.8465408805031447, 'eval_recall': 0.8476070528967254, 'eval_runtime': 110.5002, 'eval_samples_per_second': 14.109, 'eval_steps_per_second': 1.765, 'epoch': 3.0}\n",
      "{'train_runtime': 2326.4593, 'train_samples_per_second': 3.958, 'train_steps_per_second': 0.124, 'train_loss': 0.5217346615261502, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=288, training_loss=0.5217346615261502, metrics={'train_runtime': 2326.4593, 'train_samples_per_second': 3.958, 'train_steps_per_second': 0.124, 'train_loss': 0.5217346615261502, 'epoch': 3.0})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set TrainingArguments\n",
    "training_args = TrainingArguments(output_dir = \"polylm_1.7b\",\n",
    "                                  learning_rate = 2.0e-4,\n",
    "                                  warmup_steps = 32,\n",
    "                                  lr_scheduler_type = 'cosine',                                  \n",
    "                                  per_device_train_batch_size = 8,\n",
    "                                  per_device_eval_batch_size = 8,\n",
    "                                  gradient_accumulation_steps = 4,\n",
    "                                  gradient_checkpointing = True, \n",
    "                                  gradient_checkpointing_kwargs = {\"use_reentrant\": False},                                 \n",
    "                                  bf16 = True,\n",
    "                                  optim = \"paged_adamw_8bit\",                                 \n",
    "                                  num_train_epochs = 3,\n",
    "                                  weight_decay = 0.001,\n",
    "                                  evaluation_strategy = \"epoch\",\n",
    "                                  save_strategy = \"epoch\",\n",
    "                                  load_best_model_at_end = False,\n",
    "                                  metric_for_best_model = 'accuracy',\n",
    "                                  greater_is_better = True)\n",
    "\n",
    "# Set Trainer\n",
    "trainer = Trainer(model = model,\n",
    "                  args = training_args,\n",
    "                  train_dataset = ds[\"train\"],\n",
    "                  eval_dataset = ds[\"validation\"],\n",
    "                  tokenizer = tokenizer,\n",
    "                  data_collator = data_collator,\n",
    "                  compute_metrics = compute_metrics)\n",
    "\n",
    "# Train Model\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Memory Cleanup because of occasional OOM\n",
    "del trainer, model, training_args\n",
    "torch.cuda.empty_cache()\n",
    "_ = gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 OpenLLaMA 7B V2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "model_name = 'openlm-research/open_llama_7b_v2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "28a9663a507c4d7397298488eecb77f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/3069 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c237e6edfd9448719c80b28f1fd26874",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1559 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
      "        num_rows: 3069\n",
      "    })\n",
      "    validation: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
      "        num_rows: 1559\n",
      "    })\n",
      "})\n",
      "LlamaTokenizer(name_or_path='openlm-research/open_llama_7b_v2', vocab_size=32000, model_max_length=2048, is_fast=False, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'pad_token': '<unk>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={\n",
      "\t0: AddedToken(\"<unk>\", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n",
      "\t1: AddedToken(\"<s>\", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n",
      "\t2: AddedToken(\"</s>\", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Create Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name,\n",
    "                                          use_fast = False,\n",
    "                                          add_eos_token = True)\n",
    "tokenizer.pad_token_id = 0\n",
    "\n",
    "# Set Data Collator\n",
    "data_collator = DataCollatorWithPadding(tokenizer = tokenizer, padding = 'longest')\n",
    "\n",
    "# Tokenize dataset\n",
    "ds = process_dataset(tokenizer)\n",
    "\n",
    "# Summary\n",
    "print(tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reduce the memory footprint the LLM model will be quantized to 4-bits and then finetuned with a QLoRA setup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f2765dc17cc43459554ebdec28a0df8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of LlamaForSequenceClassification were not initialized from the model checkpoint at openlm-research/open_llama_7b_v2 and are newly initialized: ['score.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 33,562,624 || all params: 6,640,914,432 || trainable%: 0.5053916044795681\n",
      "PeftModelForSequenceClassification(\n",
      "  (base_model): LoraModel(\n",
      "    (model): LlamaForSequenceClassification(\n",
      "      (model): LlamaModel(\n",
      "        (embed_tokens): Embedding(32000, 4096, padding_idx=0)\n",
      "        (layers): ModuleList(\n",
      "          (0-31): 32 x LlamaDecoderLayer(\n",
      "            (self_attn): LlamaAttention(\n",
      "              (q_proj): Linear4bit(\n",
      "                (lora_dropout): ModuleDict(\n",
      "                  (default): Dropout(p=0.05, inplace=False)\n",
      "                )\n",
      "                (lora_A): ModuleDict(\n",
      "                  (default): Linear(in_features=4096, out_features=64, bias=False)\n",
      "                )\n",
      "                (lora_B): ModuleDict(\n",
      "                  (default): Linear(in_features=64, out_features=4096, bias=False)\n",
      "                )\n",
      "                (lora_embedding_A): ParameterDict()\n",
      "                (lora_embedding_B): ParameterDict()\n",
      "                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
      "              )\n",
      "              (k_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
      "              (v_proj): Linear4bit(\n",
      "                (lora_dropout): ModuleDict(\n",
      "                  (default): Dropout(p=0.05, inplace=False)\n",
      "                )\n",
      "                (lora_A): ModuleDict(\n",
      "                  (default): Linear(in_features=4096, out_features=64, bias=False)\n",
      "                )\n",
      "                (lora_B): ModuleDict(\n",
      "                  (default): Linear(in_features=64, out_features=4096, bias=False)\n",
      "                )\n",
      "                (lora_embedding_A): ParameterDict()\n",
      "                (lora_embedding_B): ParameterDict()\n",
      "                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
      "              )\n",
      "              (o_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
      "              (rotary_emb): LlamaRotaryEmbedding()\n",
      "            )\n",
      "            (mlp): LlamaMLP(\n",
      "              (gate_proj): Linear4bit(in_features=4096, out_features=11008, bias=False)\n",
      "              (up_proj): Linear4bit(in_features=4096, out_features=11008, bias=False)\n",
      "              (down_proj): Linear4bit(in_features=11008, out_features=4096, bias=False)\n",
      "              (act_fn): SiLUActivation()\n",
      "            )\n",
      "            (input_layernorm): LlamaRMSNorm()\n",
      "            (post_attention_layernorm): LlamaRMSNorm()\n",
      "          )\n",
      "        )\n",
      "        (norm): LlamaRMSNorm()\n",
      "      )\n",
      "      (score): ModulesToSaveWrapper(\n",
      "        (original_module): Linear(in_features=4096, out_features=2, bias=False)\n",
      "        (modules_to_save): ModuleDict(\n",
      "          (default): Linear(in_features=4096, out_features=2, bias=False)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Create Config\n",
    "config = AutoConfig.from_pretrained(model_name,\n",
    "                                    num_labels = 2,                                                            \n",
    "                                    use_cache = False)\n",
    "\n",
    "# Create Quantization Config\n",
    "quantization_config = BitsAndBytesConfig(load_in_4bit = True,\n",
    "                                         bnb_4bit_use_double_quant = True,\n",
    "                                         bnb_4bit_quant_type = 'nf4',\n",
    "                                         bnb_4bit_compute_dtype = torch.bfloat16)\n",
    "\n",
    "# Create Model\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name,\n",
    "                                                           config = config,\n",
    "                                                           device_map = {\"\":0},\n",
    "                                                           quantization_config = quantization_config)\n",
    "\n",
    "# Create LoRA config\n",
    "loraconfig = LoraConfig(r = 64,\n",
    "                        lora_alpha = 16,\n",
    "                        lora_dropout = 0.05,\n",
    "                        bias = 'none',\n",
    "                        task_type = TaskType.SEQ_CLS)\n",
    "\n",
    "# Prep for Training\n",
    "model = prepare_model_for_kbit_training(model)\n",
    "\n",
    "# Create LoRA Model\n",
    "model = get_peft_model(model, loraconfig)\n",
    "model.print_trainable_parameters()\n",
    "\n",
    "# Show Model Summary\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e278a30bf5914a48b1f986ace24747b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/288 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b0a5a1cce424176b5b5d3a03dc5e4e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/390 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.853     0.855     0.854       765\n",
      "           1      0.860     0.858     0.859       794\n",
      "\n",
      "    accuracy                          0.856      1559\n",
      "   macro avg      0.856     0.856     0.856      1559\n",
      "weighted avg      0.856     0.856     0.856      1559\n",
      "\n",
      "{'eval_loss': 0.3306015729904175, 'eval_accuracy': 0.8563181526619628, 'eval_precision': 0.8598484848484849, 'eval_recall': 0.8576826196473551, 'eval_runtime': 555.9299, 'eval_samples_per_second': 2.804, 'eval_steps_per_second': 0.702, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cf58d2f9f0e2404aaea671e5d0a94ce2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/390 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.957     0.762     0.849       765\n",
      "           1      0.808     0.967     0.881       794\n",
      "\n",
      "    accuracy                          0.867      1559\n",
      "   macro avg      0.883     0.865     0.865      1559\n",
      "weighted avg      0.881     0.867     0.865      1559\n",
      "\n",
      "{'eval_loss': 0.3248664438724518, 'eval_accuracy': 0.8665811417575369, 'eval_precision': 0.8084210526315789, 'eval_recall': 0.9672544080604534, 'eval_runtime': 555.936, 'eval_samples_per_second': 2.804, 'eval_steps_per_second': 0.702, 'epoch': 2.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "01cd6f70df6749d29107660e7c4cb911",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/390 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.898     0.888     0.893       765\n",
      "           1      0.893     0.903     0.898       794\n",
      "\n",
      "    accuracy                          0.895      1559\n",
      "   macro avg      0.896     0.895     0.895      1559\n",
      "weighted avg      0.895     0.895     0.895      1559\n",
      "\n",
      "{'eval_loss': 0.27301228046417236, 'eval_accuracy': 0.895445798588839, 'eval_precision': 0.8929016189290162, 'eval_recall': 0.9030226700251889, 'eval_runtime': 555.9029, 'eval_samples_per_second': 2.804, 'eval_steps_per_second': 0.702, 'epoch': 3.0}\n",
      "{'train_runtime': 11028.6779, 'train_samples_per_second': 0.835, 'train_steps_per_second': 0.026, 'train_loss': 0.33109717898898655, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=288, training_loss=0.33109717898898655, metrics={'train_runtime': 11028.6779, 'train_samples_per_second': 0.835, 'train_steps_per_second': 0.026, 'train_loss': 0.33109717898898655, 'epoch': 3.0})"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set TrainingArguments\n",
    "training_args = TrainingArguments(output_dir = \"open_llama_7b_v2\",\n",
    "                                  learning_rate = 2.0e-4,\n",
    "                                  warmup_steps = 32,\n",
    "                                  lr_scheduler_type = 'cosine',                                  \n",
    "                                  per_device_train_batch_size = 4,\n",
    "                                  per_device_eval_batch_size = 4,\n",
    "                                  gradient_accumulation_steps = 8,\n",
    "                                  gradient_checkpointing = True, \n",
    "                                  gradient_checkpointing_kwargs = {\"use_reentrant\": False},                                 \n",
    "                                  fp16 = True,\n",
    "                                  optim = \"paged_adamw_8bit\",                                 \n",
    "                                  num_train_epochs = 3,\n",
    "                                  weight_decay = 0.001,\n",
    "                                  evaluation_strategy = \"epoch\",\n",
    "                                  save_strategy = \"epoch\",\n",
    "                                  load_best_model_at_end = False,\n",
    "                                  metric_for_best_model = 'accuracy',\n",
    "                                  greater_is_better = True)\n",
    "\n",
    "# Set Trainer\n",
    "trainer = Trainer(model = model,\n",
    "                  args = training_args,\n",
    "                  train_dataset = ds[\"train\"],\n",
    "                  eval_dataset = ds[\"validation\"],\n",
    "                  tokenizer = tokenizer,\n",
    "                  data_collator = data_collator,\n",
    "                  compute_metrics = compute_metrics)\n",
    "\n",
    "# Train Model\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After training the PolyLM 1.7B model achieves an accuracy on the validation set of 84.4% while the OpenLLaMA 7B V2 model even achieves 89.5%.\n",
    "\n",
    "I did multiple training runs and on various occassions both models scored up to 0.5% higher or lower compared with the above mentioned value.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}