Transformer_Model_Training_And_Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.0 Introduction\n",
    "\n",
    "In this notebook we will train and validate 3 regular Multi-lingual Transformer models to establish a baseline of the accuracy that can be achieved when training those 3 smaller (especially small compared to current state-of-the-art LLM's) models on the earlier created training and validation CSV files.\n",
    "\n",
    "The 3 transformer models that will be used are:\n",
    "* Multi-lingual DistilBert\n",
    "* Multi-lingual Bert\n",
    "* Multi-lingual DeBERTa V3\n",
    "\n",
    "All 3 models will be trained and validated with the small data subsets.\n",
    "\n",
    "The Multi-lingual DeBERTa V3 model will also be trained on the complete dataset...just to be able to compare...\n",
    "\n",
    "These 3 transformer models consist of millions of parameters compared to billions of parameters for the GPT (and other similar LLM's) models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Modules\n",
    "import gc\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "from datasets import load_dataset, Dataset, DatasetDict\n",
    "from sklearn.metrics import classification_report, accuracy_score, precision_score, recall_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from transformers import (AutoModelForSequenceClassification, \n",
    "                          AutoTokenizer,\n",
    "                          DataCollatorWithPadding, \n",
    "                          pipeline,\n",
    "                          TrainingArguments, \n",
    "                          Trainer)\n",
    "\n",
    "# Set Seed for Randomness\n",
    "seed = 44\n",
    "torch.manual_seed(seed)\n",
    "torch.cuda.manual_seed(seed)\n",
    "torch.backends.cudnn.deterministic = False # I prefer some randomness in each training...that gives a good impression of the variations in baseline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0 Load Datasets\n",
    "\n",
    "We will reload the training and validation CSV files that were generated earlier with the notebook 'Prepare_Train_and_Validation_Datasets.ipynb'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3069, 11)\n",
      "(1559, 11)\n"
     ]
    }
   ],
   "source": [
    "# Load Datasets\n",
    "train_df = pd.read_csv('./data/train_df.csv')\n",
    "val_df = pd.read_csv('./data/val_df.csv')\n",
    "\n",
    "# Summary\n",
    "print(train_df.shape)\n",
    "print(val_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>mainSection</th>\n",
       "      <th>published_at</th>\n",
       "      <th>publisher</th>\n",
       "      <th>partisan</th>\n",
       "      <th>url</th>\n",
       "      <th>text_wordcount</th>\n",
       "      <th>max_words_text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10706318</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorlog</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorl...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2018-10-07</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/ogen-als-schoteltjes-bij-de-...</td>\n",
       "      <td>539</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorl...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12633805</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>/amsterdam</td>\n",
       "      <td>2019-05-10</td>\n",
       "      <td>parool</td>\n",
       "      <td>True</td>\n",
       "      <td>www.parool.nl/amsterdam/geen-beeld-maar-een-mo...</td>\n",
       "      <td>662</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7140125</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>/</td>\n",
       "      <td>2017-04-18</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>494</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4490774</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokikker</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokik...</td>\n",
       "      <td>/</td>\n",
       "      <td>2017-03-14</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>291</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokik...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10592180</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2018-09-11</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/meer-fouten-kabinet-bij-steu...</td>\n",
       "      <td>471</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "0  10706318   Ogen als schoteltjes bij de Tachtigjarige Oorlog   \n",
       "1  12633805  Geen beeld, maar een monument voor Mandela in ...   \n",
       "2   7140125  Hoe ga je een onveilige arbeidscultuur zoals i...   \n",
       "3   4490774  Wetenschappers ontdekken lichtgevende discokikker   \n",
       "4  10592180  Meer fouten kabinet bij steun aan strijdgroepe...   \n",
       "\n",
       "                                                text mainSection published_at  \\\n",
       "0  Ogen als schoteltjes bij de Tachtigjarige Oorl...       /home   2018-10-07   \n",
       "1  Geen beeld, maar een monument voor Mandela in ...  /amsterdam   2019-05-10   \n",
       "2  Hoe ga je een onveilige arbeidscultuur zoals i...           /   2017-04-18   \n",
       "3  Wetenschappers ontdekken lichtgevende discokik...           /   2017-03-14   \n",
       "4  Meer fouten kabinet bij steun aan strijdgroepe...       /home   2018-09-11   \n",
       "\n",
       "  publisher  partisan                                                url  \\\n",
       "0     trouw      True  www.trouw.nl/home/ogen-als-schoteltjes-bij-de-...   \n",
       "1    parool      True  www.parool.nl/amsterdam/geen-beeld-maar-een-mo...   \n",
       "2     trouw      True                                                NaN   \n",
       "3     trouw      True                                                NaN   \n",
       "4     trouw      True  www.trouw.nl/home/meer-fouten-kabinet-bij-steu...   \n",
       "\n",
       "   text_wordcount                                     max_words_text  labels  \n",
       "0             539  Ogen als schoteltjes bij de Tachtigjarige Oorl...       1  \n",
       "1             662  Geen beeld, maar een monument voor Mandela in ...       1  \n",
       "2             494  Hoe ga je een onveilige arbeidscultuur zoals i...       1  \n",
       "3             291  Wetenschappers ontdekken lichtgevende discokik...       1  \n",
       "4             471  Meer fouten kabinet bij steun aan strijdgroepe...       1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>mainSection</th>\n",
       "      <th>published_at</th>\n",
       "      <th>publisher</th>\n",
       "      <th>partisan</th>\n",
       "      <th>url</th>\n",
       "      <th>text_wordcount</th>\n",
       "      <th>max_words_text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9266995</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-12-18</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/verdachte-dodelijke-steek...</td>\n",
       "      <td>188</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4130077</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-02-11</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/honderden-arrestaties-bij...</td>\n",
       "      <td>122</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11147268</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2019-01-20</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...</td>\n",
       "      <td>262</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10749100</td>\n",
       "      <td>Klaar voor de verdediging</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-16</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/klaar-voor-de-verdediging...</td>\n",
       "      <td>411</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10700707</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-05</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/windvlaag-grijpt-springma...</td>\n",
       "      <td>286</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "0   9266995  Verdachte dodelijke steekpartijen Maastricht l...   \n",
       "1   4130077  Honderden arrestaties bij acties tegen mensen ...   \n",
       "2  11147268  Waarom de 'oudejaarsbonus' voor de jongeren va...   \n",
       "3  10749100                          Klaar voor de verdediging   \n",
       "4  10700707  Windvlaag grijpt springmatras en doodt 2-jarig...   \n",
       "\n",
       "                                                text mainSection published_at  \\\n",
       "0  Verdachte dodelijke steekpartijen Maastricht l...     /nieuws   2017-12-18   \n",
       "1  Honderden arrestaties bij acties tegen mensen ...     /nieuws   2017-02-11   \n",
       "2  Waarom de 'oudejaarsbonus' voor de jongeren va...       /home   2019-01-20   \n",
       "3  Klaar voor de verdedigingOver ruim een week be...     /nieuws   2018-10-16   \n",
       "4  Windvlaag grijpt springmatras en doodt 2-jarig...     /nieuws   2018-10-05   \n",
       "\n",
       "  publisher  partisan                                                url  \\\n",
       "0        ad     False  www.ad.nl/binnenland/verdachte-dodelijke-steek...   \n",
       "1        ad     False  www.ad.nl/buitenland/honderden-arrestaties-bij...   \n",
       "2     trouw      True  www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...   \n",
       "3        ad     False  www.ad.nl/binnenland/klaar-voor-de-verdediging...   \n",
       "4        ad     False  www.ad.nl/buitenland/windvlaag-grijpt-springma...   \n",
       "\n",
       "   text_wordcount                                     max_words_text  labels  \n",
       "0             188  Verdachte dodelijke steekpartijen Maastricht l...       0  \n",
       "1             122  Honderden arrestaties bij acties tegen mensen ...       0  \n",
       "2             262  Waarom de 'oudejaarsbonus' voor de jongeren va...       1  \n",
       "3             411  Klaar voor de verdedigingOver ruim een week be...       0  \n",
       "4             286  Windvlaag grijpt springmatras en doodt 2-jarig...       0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary\n",
    "val_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 Process Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Proces DataFrame to DataSet function.\n",
    "def process_dataset(tokenizer):\n",
    "    # Tokenize Helper\n",
    "    def preprocess_function(examples):\n",
    "        return tokenizer(examples[\"text\"], truncation = True)\n",
    "\n",
    "    # Create DataSets\n",
    "    tdf = pd.DataFrame({\"text\": train_df.max_words_text.values, \"label\": train_df.labels.values})\n",
    "    vdf = pd.DataFrame({\"text\": val_df.max_words_text.values, \"label\": val_df.labels.values})\n",
    "    tds = Dataset.from_pandas(tdf)\n",
    "    vds = Dataset.from_pandas(vdf)\n",
    "\n",
    "    ds = DatasetDict()\n",
    "    ds['train'] = tds\n",
    "    ds['validation'] = vds\n",
    "\n",
    "    # Tokenize Text\n",
    "    ds = ds.map(preprocess_function, batched = True)\n",
    "\n",
    "    # Summary\n",
    "    print(ds)\n",
    "\n",
    "    return ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.0 Evaluation Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(val_preds):\n",
    "    preds, labels = val_preds\n",
    "    preds = np.argmax(preds, axis = 1)\n",
    "    \n",
    "    report = classification_report(labels, preds, digits = 3)\n",
    "    print(report)\n",
    "    \n",
    "    accuracy_val = accuracy_score(labels, preds)\n",
    "    precision_val = precision_score(labels, preds)\n",
    "    recall_val = recall_score(labels, preds)\n",
    "    \n",
    "    return {\"accuracy\": accuracy_val, \"precision\": precision_val, \"recall\": recall_val}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.0 Training and Validation on Subset of the Data\n",
    "\n",
    "In this section we will train and validate 3 Transformer NLP models on the loaded and processed CSV files.\n",
    "\n",
    "The 3 models are the following:\n",
    "* Multi-lingual DistilBert\n",
    "* Multi-lingual Bert\n",
    "* Multi-lingual DeBERTa V3\n",
    "\n",
    "All models will be trained and validated based on the same set of hyperparameters to allow for a fair comparison.\n",
    "\n",
    "It is very likely that optimizing the hyperparameters specifically for each model could even achieve higher performance.\n",
    "\n",
    "At a later moment I will expand the notebook with some of the models being trained and validated on the complete dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set Label Info\n",
    "id2label = {0: 'NEUTRAL', 1: 'PARTISAN'}\n",
    "label2id = {'NEUTRAL': 0, 'PARTISAN': 1}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Multi-Lingual DistilBert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "model_name = 'distilbert-base-multilingual-cased'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2114377e8a6c49a895bacd464e688ec1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/3069 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3cac5345e3c24f5bb70720ede1d5794b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1559 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
      "        num_rows: 3069\n",
      "    })\n",
      "    validation: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
      "        num_rows: 1559\n",
      "    })\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "# Set Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# Data Collator\n",
    "data_collator = DataCollatorWithPadding(tokenizer = tokenizer)\n",
    "\n",
    "# Tokenize dataset\n",
    "ds = process_dataset(tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-multilingual-cased and are newly initialized: ['pre_classifier.weight', 'classifier.weight', 'pre_classifier.bias', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f20757c81454974950675c4c4e7d4b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/288 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a DistilBertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "884948c1ab2e45ce8360894f8989d439",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/49 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.870     0.788     0.827       765\n",
      "           1      0.813     0.887     0.848       794\n",
      "\n",
      "    accuracy                          0.838      1559\n",
      "   macro avg      0.842     0.837     0.838      1559\n",
      "weighted avg      0.841     0.838     0.838      1559\n",
      "\n",
      "{'eval_loss': 0.338260680437088, 'eval_accuracy': 0.8383579217447081, 'eval_precision': 0.812933025404157, 'eval_recall': 0.8866498740554156, 'eval_runtime': 4.0804, 'eval_samples_per_second': 382.068, 'eval_steps_per_second': 12.009, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c6e159fa5964d9a86008d882d918fcc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/49 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.919     0.759     0.832       765\n",
      "           1      0.802     0.936     0.863       794\n",
      "\n",
      "    accuracy                          0.849      1559\n",
      "   macro avg      0.860     0.848     0.848      1559\n",
      "weighted avg      0.859     0.849     0.848      1559\n",
      "\n",
      "{'eval_loss': 0.3530607521533966, 'eval_accuracy': 0.8492623476587556, 'eval_precision': 0.8015102481121898, 'eval_recall': 0.9357682619647355, 'eval_runtime': 4.078, 'eval_samples_per_second': 382.293, 'eval_steps_per_second': 12.016, 'epoch': 2.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "427fcbd43f184d41a8cf0bd94c7ffc88",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/49 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.855     0.851     0.853       765\n",
      "           1      0.857     0.861     0.859       794\n",
      "\n",
      "    accuracy                          0.856      1559\n",
      "   macro avg      0.856     0.856     0.856      1559\n",
      "weighted avg      0.856     0.856     0.856      1559\n",
      "\n",
      "{'eval_loss': 0.33797311782836914, 'eval_accuracy': 0.8563181526619628, 'eval_precision': 0.8571428571428571, 'eval_recall': 0.8614609571788413, 'eval_runtime': 4.0748, 'eval_samples_per_second': 382.6, 'eval_steps_per_second': 12.025, 'epoch': 3.0}\n",
      "{'train_runtime': 140.3073, 'train_samples_per_second': 65.62, 'train_steps_per_second': 2.053, 'train_loss': 0.32965970039367676, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=288, training_loss=0.32965970039367676, metrics={'train_runtime': 140.3073, 'train_samples_per_second': 65.62, 'train_steps_per_second': 2.053, 'train_loss': 0.32965970039367676, 'epoch': 3.0})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create Model\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name, \n",
    "                                                           num_labels = 2, \n",
    "                                                           id2label = id2label, \n",
    "                                                           label2id = label2id)\n",
    "model.gradient_checkpointing_enable()\n",
    "\n",
    "# Set TrainingArguments\n",
    "training_args = TrainingArguments(output_dir = \"mdistilbert\",\n",
    "                                  learning_rate = 3.0e-5,\n",
    "                                  per_device_train_batch_size = 32,\n",
    "                                  per_device_eval_batch_size = 32,\n",
    "                                  gradient_checkpointing = True, \n",
    "                                  gradient_checkpointing_kwargs = {\"use_reentrant\": False},                                 \n",
    "                                  num_train_epochs = 3,\n",
    "                                  weight_decay = 0.001,\n",
    "                                  fp16 = True,\n",
    "                                  evaluation_strategy = \"epoch\",\n",
    "                                  save_strategy = \"epoch\",\n",
    "                                  load_best_model_at_end = True,\n",
    "                                  metric_for_best_model = 'accuracy',\n",
    "                                  greater_is_better = True)\n",
    "\n",
    "trainer = Trainer(model = model,\n",
    "                  args = training_args,\n",
    "                  train_dataset = ds[\"train\"],\n",
    "                  eval_dataset = ds[\"validation\"],\n",
    "                  tokenizer = tokenizer,\n",
    "                  data_collator = data_collator,\n",
    "                  compute_metrics = compute_metrics)\n",
    "\n",
    "# Train Model\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Multi-lingual DistilBert achieves an accuracy on the validation dataset of 85.6%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Multi-Lingual Bert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "model_name = 'bert-base-multilingual-cased'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'(ReadTimeoutError(\"HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)\"), '(Request ID: 670cc97c-f016-4eed-99a2-39c097eb1ffa)')' thrown while requesting HEAD https://huggingface.co/bert-base-multilingual-cased/resolve/main/tokenizer_config.json\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f137760208c748aa836ea8aded947c30",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/3069 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "87a11c17de124ddeb045bd12e5ea5e1c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1559 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
      "        num_rows: 3069\n",
      "    })\n",
      "    validation: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
      "        num_rows: 1559\n",
      "    })\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "# Set Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# Data Collator\n",
    "data_collator = DataCollatorWithPadding(tokenizer = tokenizer)\n",
    "\n",
    "# Tokenize dataset\n",
    "ds = process_dataset(tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-multilingual-cased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "19300532b09f46bdab43c83ea7bae0cd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/288 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1c482ebf27f43b18b14b85b53a79a19",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/49 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.823     0.854     0.838       765\n",
      "           1      0.854     0.824     0.838       794\n",
      "\n",
      "    accuracy                          0.838      1559\n",
      "   macro avg      0.839     0.839     0.838      1559\n",
      "weighted avg      0.839     0.838     0.838      1559\n",
      "\n",
      "{'eval_loss': 0.3472679555416107, 'eval_accuracy': 0.8383579217447081, 'eval_precision': 0.8537859007832899, 'eval_recall': 0.8236775818639799, 'eval_runtime': 8.3434, 'eval_samples_per_second': 186.854, 'eval_steps_per_second': 5.873, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "921340bed73148d0b529974e99b4e489",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/49 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.935     0.757     0.837       765\n",
      "           1      0.802     0.950     0.870       794\n",
      "\n",
      "    accuracy                          0.855      1559\n",
      "   macro avg      0.869     0.853     0.853      1559\n",
      "weighted avg      0.868     0.855     0.853      1559\n",
      "\n",
      "{'eval_loss': 0.34125080704689026, 'eval_accuracy': 0.8550352790250161, 'eval_precision': 0.8021276595744681, 'eval_recall': 0.9496221662468514, 'eval_runtime': 8.3313, 'eval_samples_per_second': 187.126, 'eval_steps_per_second': 5.881, 'epoch': 2.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69aaa5dfac094f48b02c42a3059ec106",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/49 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.865     0.854     0.859       765\n",
      "           1      0.861     0.872     0.866       794\n",
      "\n",
      "    accuracy                          0.863      1559\n",
      "   macro avg      0.863     0.863     0.863      1559\n",
      "weighted avg      0.863     0.863     0.863      1559\n",
      "\n",
      "{'eval_loss': 0.3605136573314667, 'eval_accuracy': 0.8627325208466966, 'eval_precision': 0.8606965174129353, 'eval_recall': 0.871536523929471, 'eval_runtime': 8.3377, 'eval_samples_per_second': 186.981, 'eval_steps_per_second': 5.877, 'epoch': 3.0}\n",
      "{'train_runtime': 258.798, 'train_samples_per_second': 35.576, 'train_steps_per_second': 1.113, 'train_loss': 0.29645397928025985, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=288, training_loss=0.29645397928025985, metrics={'train_runtime': 258.798, 'train_samples_per_second': 35.576, 'train_steps_per_second': 1.113, 'train_loss': 0.29645397928025985, 'epoch': 3.0})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create Model\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name, \n",
    "                                                           num_labels = 2, \n",
    "                                                           id2label = id2label, \n",
    "                                                           label2id = label2id)\n",
    "\n",
    "# Set TrainingArguments\n",
    "training_args = TrainingArguments(output_dir = \"mbert\",\n",
    "                                  learning_rate = 3.0e-5,\n",
    "                                  per_device_train_batch_size = 16,\n",
    "                                  per_device_eval_batch_size = 32,\n",
    "                                  gradient_accumulation_steps = 2,\n",
    "                                  gradient_checkpointing = True, \n",
    "                                  gradient_checkpointing_kwargs = {\"use_reentrant\": False},                                 \n",
    "                                  num_train_epochs = 3,\n",
    "                                  weight_decay = 0.001,\n",
    "                                  fp16 = True,\n",
    "                                  evaluation_strategy = \"epoch\",\n",
    "                                  save_strategy = \"epoch\",\n",
    "                                  load_best_model_at_end = True,\n",
    "                                  metric_for_best_model = 'accuracy',\n",
    "                                  greater_is_better = True)\n",
    "\n",
    "trainer = Trainer(model = model,\n",
    "                  args = training_args,\n",
    "                  train_dataset = ds[\"train\"],\n",
    "                  eval_dataset = ds[\"validation\"],\n",
    "                  tokenizer = tokenizer,\n",
    "                  data_collator = data_collator,\n",
    "                  compute_metrics = compute_metrics)\n",
    "\n",
    "# Train Model\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Multi-lingual Bert achieves an accuracy on the validation dataset of 86.3%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Multi-Lingual DeBERTa V3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "model_name = 'microsoft/mdeberta-v3-base'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/robin/environments/LLM/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:473: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7aeca43a5d574ba893e6ebe10cd5b124",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/3069 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17475c7215c14a2a9399e5a04186b3f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1559 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
      "        num_rows: 3069\n",
      "    })\n",
      "    validation: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
      "        num_rows: 1559\n",
      "    })\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "# Set Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# Data Collator\n",
    "data_collator = DataCollatorWithPadding(tokenizer = tokenizer)\n",
    "\n",
    "# Tokenize dataset\n",
    "ds = process_dataset(tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of DebertaV2ForSequenceClassification were not initialized from the model checkpoint at microsoft/mdeberta-v3-base and are newly initialized: ['classifier.weight', 'pooler.dense.bias', 'pooler.dense.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "759c1b2cfe164ebcad7c6712f615ba89",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/288 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a DebertaV2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5a629355fba0491a815b22ccb704d44e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/49 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.855     0.861     0.858       765\n",
      "           1      0.865     0.859     0.862       794\n",
      "\n",
      "    accuracy                          0.860      1559\n",
      "   macro avg      0.860     0.860     0.860      1559\n",
      "weighted avg      0.860     0.860     0.860      1559\n",
      "\n",
      "{'eval_loss': 0.3431350290775299, 'eval_accuracy': 0.8601667735728031, 'eval_precision': 0.8654822335025381, 'eval_recall': 0.8589420654911839, 'eval_runtime': 14.9399, 'eval_samples_per_second': 104.351, 'eval_steps_per_second': 3.28, 'epoch': 1.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4030a4da542473cb422e128279de262",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/49 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.758     0.954     0.845       765\n",
      "           1      0.941     0.707     0.807       794\n",
      "\n",
      "    accuracy                          0.828      1559\n",
      "   macro avg      0.850     0.830     0.826      1559\n",
      "weighted avg      0.851     0.828     0.826      1559\n",
      "\n",
      "{'eval_loss': 0.4065735340118408, 'eval_accuracy': 0.828094932649134, 'eval_precision': 0.9412751677852349, 'eval_recall': 0.7065491183879093, 'eval_runtime': 14.9392, 'eval_samples_per_second': 104.356, 'eval_steps_per_second': 3.28, 'epoch': 2.0}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b9dc4a6f07240f4a58cbb396730d31c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/49 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.805     0.940     0.867       765\n",
      "           1      0.931     0.781     0.849       794\n",
      "\n",
      "    accuracy                          0.859      1559\n",
      "   macro avg      0.868     0.860     0.858      1559\n",
      "weighted avg      0.869     0.859     0.858      1559\n",
      "\n",
      "{'eval_loss': 0.3701871931552887, 'eval_accuracy': 0.8588838999358563, 'eval_precision': 0.9309309309309309, 'eval_recall': 0.7808564231738035, 'eval_runtime': 14.9436, 'eval_samples_per_second': 104.326, 'eval_steps_per_second': 3.279, 'epoch': 3.0}\n",
      "{'train_runtime': 479.9446, 'train_samples_per_second': 19.183, 'train_steps_per_second': 0.6, 'train_loss': 0.33482729064093697, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=288, training_loss=0.33482729064093697, metrics={'train_runtime': 479.9446, 'train_samples_per_second': 19.183, 'train_steps_per_second': 0.6, 'train_loss': 0.33482729064093697, 'epoch': 3.0})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create Model\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name, \n",
    "                                                           num_labels = 2, \n",
    "                                                           id2label = id2label, \n",
    "                                                           label2id = label2id)\n",
    "\n",
    "# Set TrainingArguments\n",
    "training_args = TrainingArguments(output_dir = \"mdebertav3\",\n",
    "                                  learning_rate = 3.0e-5,\n",
    "                                  per_device_train_batch_size = 16,\n",
    "                                  per_device_eval_batch_size = 32,\n",
    "                                  gradient_accumulation_steps = 2,\n",
    "                                  gradient_checkpointing = True, \n",
    "                                  gradient_checkpointing_kwargs = {\"use_reentrant\": False},\n",
    "                                  num_train_epochs = 3,\n",
    "                                  weight_decay = 0.001,\n",
    "                                  fp16 = True,\n",
    "                                  evaluation_strategy = \"epoch\",\n",
    "                                  save_strategy = \"epoch\",\n",
    "                                  load_best_model_at_end = True,\n",
    "                                  metric_for_best_model = 'accuracy',\n",
    "                                  greater_is_better = True)\n",
    "\n",
    "trainer = Trainer(model = model,\n",
    "                  args = training_args,\n",
    "                  train_dataset = ds[\"train\"],\n",
    "                  eval_dataset = ds[\"validation\"],\n",
    "                  tokenizer = tokenizer,\n",
    "                  data_collator = data_collator,\n",
    "                  compute_metrics = compute_metrics)\n",
    "\n",
    "# Train Model\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Memory Cleanup because of occasional OOM\n",
    "del ds, model, trainer, training_args\n",
    "torch.cuda.empty_cache()\n",
    "_ = gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Multi-lingual DeBERTa V3 achieves an accuracy on the validation dataset of 85.8%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.0 Multi-Lingual DeBERTa V3 - Train and Validate with complete dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the Multi-lingual DeBERTa v3 is the newest of the 3 transformer models we will use this model again for a test with the complete dataset.\n",
    "Feel free to run this test yourself for any other model.\n",
    "\n",
    "The complete dataset will be used with 80% for training and the remaining 20% of the data for validation. All other things will be equal.\n",
    "\n",
    "First the complete dataset will be processed. For a simple EDA and further explanation of the code please look at the notebook 'Prepare_Train_and_Validation_Datasets.ipynb'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DPGNews2019 Dataframe Shape: (103870, 8)\n",
      "Training Dataset Shape: (83096, 10)\n",
      "Validation Dataset Shape: (20774, 10)\n"
     ]
    }
   ],
   "source": [
    "# Constants\n",
    "SEED = 42\n",
    "MAX_WORDS = 192\n",
    "\n",
    "def get_dpgnews_df(seed):\n",
    "    # Set 1: Articles\n",
    "    articles_df = pd.read_json('./dpgMedia2019-articles-bypublisher.jsonl', lines = True)\n",
    "    articles_df = articles_df.set_index('id')\n",
    "    \n",
    "    # Set 2: Labels\n",
    "    labels_df = pd.read_json('./dpgMedia2019-labels-bypublisher.jsonl', lines = True)\n",
    "    labels_df = labels_df.set_index('id')\n",
    "    \n",
    "    # Finalize Full Data\n",
    "    dpgnews_df = articles_df.join(labels_df, on = ['id'], how = 'inner')\n",
    "    \n",
    "    # Randomize all rows...\n",
    "    dpgnews_df = dpgnews_df.sample(frac = 1.0, random_state = seed)\n",
    "    dpgnews_df.reset_index(inplace = True)\n",
    "    print(f'DPGNews2019 Dataframe Shape: {dpgnews_df.shape}') \n",
    "\n",
    "    return dpgnews_df\n",
    "\n",
    "# Maximize Text to number of words\n",
    "def maximize_word_count(text):\n",
    "    text_list = text.split(' ')\n",
    "    \n",
    "    if len(text_list) >= MAX_WORDS:\n",
    "        maximized_text = ' '.join(text_list[:MAX_WORDS])\n",
    "    else:\n",
    "        maximized_text = ' '.join(text_list)\n",
    "    return maximized_text\n",
    "\n",
    "# Get DpgNews Dataframe\n",
    "dpgnews_df = get_dpgnews_df(SEED)\n",
    "\n",
    "# Map\n",
    "dpgnews_df['max_words_text'] =  dpgnews_df.apply(lambda x: maximize_word_count(x.text), axis = 1)   \n",
    "\n",
    "# Partisan Modify\n",
    "labels = []\n",
    "\n",
    "# Tokenize\n",
    "for index, row in dpgnews_df.iterrows():\n",
    "    partisan = row['partisan']\n",
    "    labels.append(1 if partisan == 'true' else 0)\n",
    "dpgnews_df[\"labels\"] = labels\n",
    "\n",
    "# Train Test Split\n",
    "train_df, val_df = train_test_split(dpgnews_df, \n",
    "                                    test_size = 0.20, \n",
    "                                    random_state = SEED,\n",
    "                                    stratify = dpgnews_df.partisan.values)\n",
    "\n",
    "# Summary\n",
    "print(f'Training Dataset Shape: {train_df.shape}')\n",
    "print(f'Validation Dataset Shape: {val_df.shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "model_name = 'microsoft/mdeberta-v3-base'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/robin/environments/LLM/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:473: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c9b8632183dd4f2e9fafd1e179fe4977",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/83096 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4adeb981185451095331106ca1c0929",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/20774 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
      "        num_rows: 83096\n",
      "    })\n",
      "    validation: Dataset({\n",
      "        features: ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
      "        num_rows: 20774\n",
      "    })\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "# Set Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# Data Collator\n",
    "data_collator = DataCollatorWithPadding(tokenizer = tokenizer)\n",
    "\n",
    "# Tokenize dataset\n",
    "ds = process_dataset(tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of DebertaV2ForSequenceClassification were not initialized from the model checkpoint at microsoft/mdeberta-v3-base and are newly initialized: ['classifier.weight', 'pooler.dense.bias', 'pooler.dense.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69dc8bbb522a40f993dbbbd6032b58f6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7791 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a DebertaV2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.335, 'learning_rate': 2.8074701578744707e-05, 'epoch': 0.19}\n",
      "{'loss': 0.2226, 'learning_rate': 2.6157104351174434e-05, 'epoch': 0.39}\n",
      "{'loss': 0.1971, 'learning_rate': 2.4231805929919137e-05, 'epoch': 0.58}\n",
      "{'loss': 0.1786, 'learning_rate': 2.2306507508663844e-05, 'epoch': 0.77}\n",
      "{'loss': 0.1667, 'learning_rate': 2.0381209087408547e-05, 'epoch': 0.96}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f3386e85cb34cb4879ab3d72a95c0b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/650 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.920     0.961     0.940     10190\n",
      "           1      0.961     0.920     0.940     10584\n",
      "\n",
      "    accuracy                          0.940     20774\n",
      "   macro avg      0.941     0.940     0.940     20774\n",
      "weighted avg      0.941     0.940     0.940     20774\n",
      "\n",
      "{'eval_loss': 0.164809450507164, 'eval_accuracy': 0.9400693174160007, 'eval_precision': 0.9611851851851851, 'eval_recall': 0.9195011337868481, 'eval_runtime': 201.3636, 'eval_samples_per_second': 103.167, 'eval_steps_per_second': 3.228, 'epoch': 1.0}\n",
      "{'loss': 0.1422, 'learning_rate': 1.8455910666153254e-05, 'epoch': 1.16}\n",
      "{'loss': 0.1325, 'learning_rate': 1.653446284174047e-05, 'epoch': 1.35}\n",
      "{'loss': 0.1232, 'learning_rate': 1.4609164420485175e-05, 'epoch': 1.54}\n",
      "{'loss': 0.1224, 'learning_rate': 1.268386599922988e-05, 'epoch': 1.73}\n",
      "{'loss': 0.1195, 'learning_rate': 1.0758567577974585e-05, 'epoch': 1.93}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "641d98cd1c204e98a0285923f710a9a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/650 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.935     0.964     0.949     10190\n",
      "           1      0.964     0.935     0.950     10584\n",
      "\n",
      "    accuracy                          0.949     20774\n",
      "   macro avg      0.950     0.950     0.949     20774\n",
      "weighted avg      0.950     0.949     0.949     20774\n",
      "\n",
      "{'eval_loss': 0.14764031767845154, 'eval_accuracy': 0.9493597766438818, 'eval_precision': 0.9642509253847652, 'eval_recall': 0.9352796674225246, 'eval_runtime': 201.2591, 'eval_samples_per_second': 103.22, 'eval_steps_per_second': 3.23, 'epoch': 2.0}\n",
      "{'loss': 0.0993, 'learning_rate': 8.833269156719292e-06, 'epoch': 2.12}\n",
      "{'loss': 0.0839, 'learning_rate': 6.911821332306508e-06, 'epoch': 2.31}\n",
      "{'loss': 0.0829, 'learning_rate': 4.9865229110512135e-06, 'epoch': 2.5}\n",
      "{'loss': 0.081, 'learning_rate': 3.065075086638429e-06, 'epoch': 2.7}\n",
      "{'loss': 0.0783, 'learning_rate': 1.1397766653831344e-06, 'epoch': 2.89}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a911e13767948f6b1f714dc65a62b58",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/650 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0      0.937     0.967     0.952     10190\n",
      "           1      0.968     0.937     0.952     10584\n",
      "\n",
      "    accuracy                          0.952     20774\n",
      "   macro avg      0.952     0.952     0.952     20774\n",
      "weighted avg      0.952     0.952     0.952     20774\n",
      "\n",
      "{'eval_loss': 0.17834103107452393, 'eval_accuracy': 0.9518147684605757, 'eval_precision': 0.9675090252707581, 'eval_recall': 0.936885865457294, 'eval_runtime': 200.8521, 'eval_samples_per_second': 103.429, 'eval_steps_per_second': 3.236, 'epoch': 3.0}\n",
      "{'train_runtime': 11725.4649, 'train_samples_per_second': 21.26, 'train_steps_per_second': 0.664, 'train_loss': 0.1417716857083663, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=7791, training_loss=0.1417716857083663, metrics={'train_runtime': 11725.4649, 'train_samples_per_second': 21.26, 'train_steps_per_second': 0.664, 'train_loss': 0.1417716857083663, 'epoch': 3.0})"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create Model\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name, \n",
    "                                                           num_labels = 2, \n",
    "                                                           id2label = id2label, \n",
    "                                                           label2id = label2id)\n",
    "\n",
    "# Set TrainingArguments\n",
    "training_args = TrainingArguments(output_dir = \"mdebertav3\",\n",
    "                                  learning_rate = 3.0e-5,\n",
    "                                  per_device_train_batch_size = 16,\n",
    "                                  per_device_eval_batch_size = 32,\n",
    "                                  gradient_accumulation_steps = 2,\n",
    "                                  gradient_checkpointing = True, \n",
    "                                  gradient_checkpointing_kwargs = {\"use_reentrant\": False},\n",
    "                                  num_train_epochs = 3,\n",
    "                                  weight_decay = 0.001,\n",
    "                                  fp16 = True,\n",
    "                                  evaluation_strategy = \"epoch\",\n",
    "                                  save_strategy = \"epoch\",\n",
    "                                  load_best_model_at_end = True,\n",
    "                                  metric_for_best_model = 'accuracy',\n",
    "                                  greater_is_better = True)\n",
    "\n",
    "trainer = Trainer(model = model,\n",
    "                  args = training_args,\n",
    "                  train_dataset = ds[\"train\"],\n",
    "                  eval_dataset = ds[\"validation\"],\n",
    "                  tokenizer = tokenizer,\n",
    "                  data_collator = data_collator,\n",
    "                  compute_metrics = compute_metrics)\n",
    "\n",
    "# Train Model\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Multi-lingual DeBERTa V3 achieves an accuracy on the validation dataset of 95.2% after training on 80% of the full dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.0 Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the 3 multi-lingual Transformer NLP models trained and validated we have an interresting baseline to compare with a finetuned GPT-3.5/GPT-4 model.\n",
    "\n",
    "With an achieved accuray of 85.6% for the DistilBert model, an achieved accuracy of 86.3% for the Bert model and an achieved accuracy of 85.8% for the DeBERTa V3 model it will be very interresting to see if we can finetune a GPT model as classifier and achieve/exceed the 86% - 88% accuracy target as achieved by the best regular model.\n",
    "\n",
    "I did multiple training runs and on various occassions the models scored up to 2% higher or 1% lower compared with the above mentioned values. \n",
    "\n",
    "The Multi-lingual DeBERTa V3 achieves an accuracy on the validation dataset of 95.2% after training on 80% of the full dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}