Finetune_GPT-3.5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.0 Introduction\n",
    "\n",
    "In this notebook we will perform all steps required to finetune a GPT-3.5 Turbo model on our own custom datasets.\n",
    "\n",
    "The training and validation csv files are the same ones that were also used in the notebook 'Transformer_Model_Training_And_Validation.ipynb' where we trained a multi-lingual DistilBert, Bert and DeBERTa V3 model.\n",
    "\n",
    "After the GPT-3.5 Turbo model is fine-tuned and validated we can compare the performance on the validation set across the various models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Updated in latest version\n",
    "\n",
    "In this latest version (December 5th, 2023) of this notebook I have made the following updates:\n",
    "* Updated to the latest version of OpenAI (1.3.7) and modified the API calls accordingly.\n",
    "* Changed the model to the latest version: \"gpt-3.5-turbo-1106\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Modules\n",
    "import json\n",
    "import os\n",
    "from io import StringIO\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# OpenAI\n",
    "from openai import OpenAI\n",
    "import tiktoken # for token counting\n",
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0 Load Datasets\n",
    "\n",
    "We will load the training and validation CSV files that were generated earlier with the notebook 'Prepare_Train_and_Validation_Datasets.ipynb'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3069, 11)\n",
      "(1559, 11)\n"
     ]
    }
   ],
   "source": [
    "# Load Datasets\n",
    "train_df = pd.read_csv('./data/train_df.csv')\n",
    "val_df = pd.read_csv('./data/val_df.csv')\n",
    "\n",
    "# Summary\n",
    "print(train_df.shape)\n",
    "print(val_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's review a small subset of the training data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>mainSection</th>\n",
       "      <th>published_at</th>\n",
       "      <th>publisher</th>\n",
       "      <th>partisan</th>\n",
       "      <th>url</th>\n",
       "      <th>text_wordcount</th>\n",
       "      <th>max_words_text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10706318</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorlog</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorl...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2018-10-07</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/ogen-als-schoteltjes-bij-de-...</td>\n",
       "      <td>539</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorl...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12633805</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>/amsterdam</td>\n",
       "      <td>2019-05-10</td>\n",
       "      <td>parool</td>\n",
       "      <td>True</td>\n",
       "      <td>www.parool.nl/amsterdam/geen-beeld-maar-een-mo...</td>\n",
       "      <td>662</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7140125</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>/</td>\n",
       "      <td>2017-04-18</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>494</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4490774</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokikker</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokik...</td>\n",
       "      <td>/</td>\n",
       "      <td>2017-03-14</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>291</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokik...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10592180</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2018-09-11</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/meer-fouten-kabinet-bij-steu...</td>\n",
       "      <td>471</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "0  10706318   Ogen als schoteltjes bij de Tachtigjarige Oorlog   \n",
       "1  12633805  Geen beeld, maar een monument voor Mandela in ...   \n",
       "2   7140125  Hoe ga je een onveilige arbeidscultuur zoals i...   \n",
       "3   4490774  Wetenschappers ontdekken lichtgevende discokikker   \n",
       "4  10592180  Meer fouten kabinet bij steun aan strijdgroepe...   \n",
       "\n",
       "                                                text mainSection published_at  \\\n",
       "0  Ogen als schoteltjes bij de Tachtigjarige Oorl...       /home   2018-10-07   \n",
       "1  Geen beeld, maar een monument voor Mandela in ...  /amsterdam   2019-05-10   \n",
       "2  Hoe ga je een onveilige arbeidscultuur zoals i...           /   2017-04-18   \n",
       "3  Wetenschappers ontdekken lichtgevende discokik...           /   2017-03-14   \n",
       "4  Meer fouten kabinet bij steun aan strijdgroepe...       /home   2018-09-11   \n",
       "\n",
       "  publisher  partisan                                                url  \\\n",
       "0     trouw      True  www.trouw.nl/home/ogen-als-schoteltjes-bij-de-...   \n",
       "1    parool      True  www.parool.nl/amsterdam/geen-beeld-maar-een-mo...   \n",
       "2     trouw      True                                                NaN   \n",
       "3     trouw      True                                                NaN   \n",
       "4     trouw      True  www.trouw.nl/home/meer-fouten-kabinet-bij-steu...   \n",
       "\n",
       "   text_wordcount                                     max_words_text  labels  \n",
       "0             539  Ogen als schoteltjes bij de Tachtigjarige Oorl...       1  \n",
       "1             662  Geen beeld, maar een monument voor Mandela in ...       1  \n",
       "2             494  Hoe ga je een onveilige arbeidscultuur zoals i...       1  \n",
       "3             291  Wetenschappers ontdekken lichtgevende discokik...       1  \n",
       "4             471  Meer fouten kabinet bij steun aan strijdgroepe...       1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and also the validation data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>mainSection</th>\n",
       "      <th>published_at</th>\n",
       "      <th>publisher</th>\n",
       "      <th>partisan</th>\n",
       "      <th>url</th>\n",
       "      <th>text_wordcount</th>\n",
       "      <th>max_words_text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9266995</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-12-18</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/verdachte-dodelijke-steek...</td>\n",
       "      <td>188</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4130077</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-02-11</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/honderden-arrestaties-bij...</td>\n",
       "      <td>122</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11147268</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2019-01-20</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...</td>\n",
       "      <td>262</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10749100</td>\n",
       "      <td>Klaar voor de verdediging</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-16</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/klaar-voor-de-verdediging...</td>\n",
       "      <td>411</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10700707</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-05</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/windvlaag-grijpt-springma...</td>\n",
       "      <td>286</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "0   9266995  Verdachte dodelijke steekpartijen Maastricht l...   \n",
       "1   4130077  Honderden arrestaties bij acties tegen mensen ...   \n",
       "2  11147268  Waarom de 'oudejaarsbonus' voor de jongeren va...   \n",
       "3  10749100                          Klaar voor de verdediging   \n",
       "4  10700707  Windvlaag grijpt springmatras en doodt 2-jarig...   \n",
       "\n",
       "                                                text mainSection published_at  \\\n",
       "0  Verdachte dodelijke steekpartijen Maastricht l...     /nieuws   2017-12-18   \n",
       "1  Honderden arrestaties bij acties tegen mensen ...     /nieuws   2017-02-11   \n",
       "2  Waarom de 'oudejaarsbonus' voor de jongeren va...       /home   2019-01-20   \n",
       "3  Klaar voor de verdedigingOver ruim een week be...     /nieuws   2018-10-16   \n",
       "4  Windvlaag grijpt springmatras en doodt 2-jarig...     /nieuws   2018-10-05   \n",
       "\n",
       "  publisher  partisan                                                url  \\\n",
       "0        ad     False  www.ad.nl/binnenland/verdachte-dodelijke-steek...   \n",
       "1        ad     False  www.ad.nl/buitenland/honderden-arrestaties-bij...   \n",
       "2     trouw      True  www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...   \n",
       "3        ad     False  www.ad.nl/binnenland/klaar-voor-de-verdediging...   \n",
       "4        ad     False  www.ad.nl/buitenland/windvlaag-grijpt-springma...   \n",
       "\n",
       "   text_wordcount                                     max_words_text  labels  \n",
       "0             188  Verdachte dodelijke steekpartijen Maastricht l...       0  \n",
       "1             122  Honderden arrestaties bij acties tegen mensen ...       0  \n",
       "2             262  Waarom de 'oudejaarsbonus' voor de jongeren va...       1  \n",
       "3             411  Klaar voor de verdedigingOver ruim een week be...       0  \n",
       "4             286  Windvlaag grijpt springmatras en doodt 2-jarig...       0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary\n",
    "val_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 Finetuning OpenAI GPT-3.5 Model\n",
    "\n",
    "In this section we will proces and upload the files for training and validation to OpenAI.\n",
    "\n",
    "After the files are uploaded we can create a fine-tuning job on OpenAI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "MAX_WORDS = 192\n",
    "\n",
    "# OpenAI API Key\n",
    "client = OpenAI(api_key = os.environ[\"OPENAI_API_KEY\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Create and Validate OpenAI Files\n",
    "\n",
    "Part of creating the required files is engineering a prompt that matches what we want the model to perform with the finetuning.\n",
    "\n",
    "In the earlier notebook we trained the 2 classical Transformer models to classify the input text as either partisan or neutral.\n",
    "\n",
    "With our prompt we want to achieve the same. As part of the prompt's system message we tell GPT-3.5 that it is a newspaper editor and that it needs to classify each newspaper article as being partisan or neutral.\n",
    "\n",
    "The news article is then added as part of the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_prompt(item_text, item_label = None, inference = False):\n",
    "    if inference:\n",
    "        # Base Prompt\n",
    "        prompt_text = [{\"role\": \"system\", \"content\": \"Je bent redacteur bij een krant. Je beoordeeld een krantenartikel of het politiek of neutraal is. Hieronder staat de tekst van het krantenartikel.\"}, {\"role\": \"user\", \"content\": \"\"}]\n",
    "\n",
    "        # Set Text and Label\n",
    "        prompt_text[1]['content'] = '### Tekst:\\n' + item_text\n",
    "    else:   \n",
    "        # Base Prompt\n",
    "        prompt_text = {\"messages\": [{\"role\": \"system\", \"content\": \"Je bent redacteur bij een krant. Je beoordeeld een krantenartikel of het politiek of neutraal is. Hieronder staat de tekst van het krantenartikel.\"}, {\"role\": \"user\", \"content\": \"\"}, {\"role\": \"assistant\", \"content\": \"\"}]}\n",
    "\n",
    "        # Set Text and Label\n",
    "        prompt_text['messages'][1]['content'] = '### Tekst:\\n' + item_text\n",
    "        prompt_text['messages'][2]['content'] = item_label\n",
    "    \n",
    "    return prompt_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function 'create_openai_file' creates the JSON files for OpenAI based on the input Pandas DataFrame. For each row a prompt is generated and the 'label' for finetuning is set.\n",
    "\n",
    "As the dataset is in the Dutch language the labels are specified as either 'Politiek' (roughly equals partisan) or 'Neutraal' (neutral).\n",
    "\n",
    "Note that we use the 'max_words_text' column to make sure that each prompt uses the same text as was used when training the Multi-lingual DistilBert and Bert models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_openai_file(df, file_name):\n",
    "    # Create Train JSON File\n",
    "    jsonl_file = []\n",
    "\n",
    "    # Loop through rows in Pandas DataFrame\n",
    "    for index, row in df.iterrows():\n",
    "        text = row['max_words_text']\n",
    "        partisan = row['partisan']\n",
    "\n",
    "        if partisan == True:\n",
    "            label = 'Politiek'\n",
    "        else:\n",
    "            label = 'Neutraal'\n",
    "\n",
    "        jsonl_file.append(create_prompt(text, label))\n",
    "\n",
    "    # Save to File\n",
    "    with open(file_name, 'w') as out_file:\n",
    "        for item in jsonl_file:        \n",
    "            out_file.write(json.dumps(item) + '\\n')\n",
    "\n",
    "    # Summary\n",
    "    print(f'\\n======== {file_name}')\n",
    "    print(f'Length Messages: {len(jsonl_file)}')\n",
    "    print(jsonl_file[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code to validate the OpenAI files is based on and combined from the OpenAI Cookbook: https://cookbook.openai.com/examples/chat_finetuning_data_prep\n",
    "\n",
    "Only minor updates have been made for this specific example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validate OpenAI file\n",
    "def validate_openai_file(data_path):\n",
    "    print('\\n======= OpenAI Validation')\n",
    "    \n",
    "    # Load the dataset\n",
    "    with open(data_path, 'r', encoding='utf-8') as f:\n",
    "        dataset = [json.loads(line) for line in f]\n",
    "\n",
    "    # Initial dataset stats\n",
    "    print(\"Num examples:\", len(dataset))\n",
    "    print(\"First example:\")\n",
    "    for message in dataset[0][\"messages\"]:\n",
    "        print(message)\n",
    "\n",
    "    # Format error checks\n",
    "    format_errors = defaultdict(int)\n",
    "\n",
    "    for ex in dataset:\n",
    "        if not isinstance(ex, dict):\n",
    "            format_errors[\"data_type\"] += 1\n",
    "            continue\n",
    "            \n",
    "        messages = ex.get(\"messages\", None)\n",
    "        if not messages:\n",
    "            format_errors[\"missing_messages_list\"] += 1\n",
    "            continue\n",
    "            \n",
    "        for message in messages:\n",
    "            if \"role\" not in message or \"content\" not in message:\n",
    "                format_errors[\"message_missing_key\"] += 1\n",
    "            \n",
    "            if any(k not in (\"role\", \"content\", \"name\", \"function_call\") for k in message):\n",
    "                format_errors[\"message_unrecognized_key\"] += 1\n",
    "            \n",
    "            if message.get(\"role\", None) not in (\"system\", \"user\", \"assistant\", \"function\"):\n",
    "                format_errors[\"unrecognized_role\"] += 1\n",
    "                \n",
    "            content = message.get(\"content\", None)\n",
    "            function_call = message.get(\"function_call\", None)\n",
    "            \n",
    "            if (not content and not function_call) or not isinstance(content, str):\n",
    "                format_errors[\"missing_content\"] += 1\n",
    "        \n",
    "        if not any(message.get(\"role\", None) == \"assistant\" for message in messages):\n",
    "            format_errors[\"example_missing_assistant_message\"] += 1\n",
    "\n",
    "    if format_errors:\n",
    "        print(\"Found errors:\")\n",
    "        for k, v in format_errors.items():\n",
    "            print(f\"{k}: {v}\")\n",
    "    else:\n",
    "        print(\"No errors found\")\n",
    "        \n",
    "    encoding = tiktoken.get_encoding(\"cl100k_base\")\n",
    "\n",
    "    # not exact!\n",
    "    # simplified from https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb\n",
    "    def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1):\n",
    "        num_tokens = 0\n",
    "        for message in messages:\n",
    "            num_tokens += tokens_per_message\n",
    "            for key, value in message.items():\n",
    "                num_tokens += len(encoding.encode(value))\n",
    "                if key == \"name\":\n",
    "                    num_tokens += tokens_per_name\n",
    "        num_tokens += 3\n",
    "        return num_tokens\n",
    "\n",
    "    def num_assistant_tokens_from_messages(messages):\n",
    "        num_tokens = 0\n",
    "        for message in messages:\n",
    "            if message[\"role\"] == \"assistant\":\n",
    "                num_tokens += len(encoding.encode(message[\"content\"]))\n",
    "        return num_tokens\n",
    "\n",
    "    def print_distribution(values, name):\n",
    "        print(f\"\\n#### Distribution of {name}:\")\n",
    "        print(f\"min / max: {min(values)}, {max(values)}\")\n",
    "        print(f\"mean / median: {np.mean(values)}, {np.median(values)}\")\n",
    "        print(f\"p5 / p95: {np.quantile(values, 0.1)}, {np.quantile(values, 0.9)}\")\n",
    "        \n",
    "    # Warnings and tokens counts\n",
    "    n_missing_system = 0\n",
    "    n_missing_user = 0\n",
    "    n_messages = []\n",
    "    convo_lens = []\n",
    "    assistant_message_lens = []\n",
    "\n",
    "    for ex in dataset:\n",
    "        messages = ex[\"messages\"]\n",
    "        if not any(message[\"role\"] == \"system\" for message in messages):\n",
    "            n_missing_system += 1\n",
    "        if not any(message[\"role\"] == \"user\" for message in messages):\n",
    "            n_missing_user += 1\n",
    "        n_messages.append(len(messages))\n",
    "        convo_lens.append(num_tokens_from_messages(messages))\n",
    "        assistant_message_lens.append(num_assistant_tokens_from_messages(messages))\n",
    "        \n",
    "    print(\"Num examples missing system message:\", n_missing_system)\n",
    "    print(\"Num examples missing user message:\", n_missing_user)\n",
    "    print_distribution(n_messages, \"num_messages_per_example\")\n",
    "    print_distribution(convo_lens, \"num_total_tokens_per_example\")\n",
    "    print_distribution(assistant_message_lens, \"num_assistant_tokens_per_example\")\n",
    "    n_too_long = sum(l > 4096 for l in convo_lens)\n",
    "    print(f\"\\n{n_too_long} examples may be over the 4096 token limit, they will be truncated during fine-tuning\")\n",
    "\n",
    "    # Pricing and default n_epochs estimate\n",
    "    MAX_TOKENS_PER_EXAMPLE = 4096\n",
    "\n",
    "    TARGET_EPOCHS = 1\n",
    "    MIN_TARGET_EXAMPLES = 100\n",
    "    MAX_TARGET_EXAMPLES = 25000\n",
    "    MIN_DEFAULT_EPOCHS = 1\n",
    "    MAX_DEFAULT_EPOCHS = 2\n",
    "\n",
    "    n_epochs = TARGET_EPOCHS\n",
    "    n_train_examples = len(dataset)\n",
    "    if n_train_examples * TARGET_EPOCHS < MIN_TARGET_EXAMPLES:\n",
    "        n_epochs = min(MAX_DEFAULT_EPOCHS, MIN_TARGET_EXAMPLES // n_train_examples)\n",
    "    elif n_train_examples * TARGET_EPOCHS > MAX_TARGET_EXAMPLES:\n",
    "        n_epochs = max(MIN_DEFAULT_EPOCHS, MAX_TARGET_EXAMPLES // n_train_examples)\n",
    "\n",
    "    n_billing_tokens_in_dataset = sum(min(MAX_TOKENS_PER_EXAMPLE, length) for length in convo_lens)\n",
    "    print(f\"Dataset has ~{n_billing_tokens_in_dataset} tokens that will be charged for during training\")\n",
    "    print(f\"By default, you'll train for {n_epochs} epochs on this dataset\")\n",
    "    print(f\"By default, you'll be charged for ~{n_epochs * n_billing_tokens_in_dataset} tokens\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First the training file for OpenAI fine-tuning is created locally and validated with the code from the OpenAI Cookbook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======== ./data/train_192_v1.jsonl\n",
      "Length Messages: 3069\n",
      "[{'messages': [{'role': 'system', 'content': 'Je bent redacteur bij een krant. Je beoordeeld een krantenartikel of het politiek of neutraal is. Hieronder staat de tekst van het krantenartikel.'}, {'role': 'user', 'content': \"### Tekst:\\nOgen als schoteltjes bij de Tachtigjarige Oorlog Het was mijn favoriete oorlog op de basisschool. Tachtig jaar vechten? Onvoorstelbaar. Dat we met Willem van Oranje wonnen van die in de waterlinies verzuipende Spanjaarden, en dat die wrede Alva het nakijken had, was helemaal mooi. Zielig dat Van Oranje dood moest. Maar ja, ons land was nu wel ontstaan.Zo ongeveer herinner ik me het. En nu zat ik zondagmiddag met mijn eigen kinderen te kijken naar de NTR-jeugdserie 'Welkom in de 80-jarige Oorlog', 450 jaar na de eerste gevechten in 1568. Ik was benieuwd welke geschiedversie zij te zien zouden krijgen.In de volwassenenversie '80 Jaar Oorlog' legt Hans Goedkoop een bom onder het beeld van goede protestanten tegen slechte katholieken uit Spanje. Het was eerder een burgeroorlog. De 'heldhaftige' geuzen waren piraten die plunderden en moordden. En de protestanten met hun beeldenstorm leken op de geloofsfanatici van IS en de Taliban, zeiden de historici.Zat ik dus kritisch te kijken hoe die visie zou doorsijpelen in de kindervariant, zag ik naast mij ogen als schoteltjes. Mijn dochter (8) en zoon (7) vergaapten zich eraan. Die kostuums! Die sfeer van opstand, intrige en macht. En\"}, {'role': 'assistant', 'content': 'Politiek'}]}]\n",
      "\n",
      "======= OpenAI Validation\n",
      "Num examples: 3069\n",
      "First example:\n",
      "{'role': 'system', 'content': 'Je bent redacteur bij een krant. Je beoordeeld een krantenartikel of het politiek of neutraal is. Hieronder staat de tekst van het krantenartikel.'}\n",
      "{'role': 'user', 'content': \"### Tekst:\\nOgen als schoteltjes bij de Tachtigjarige Oorlog Het was mijn favoriete oorlog op de basisschool. Tachtig jaar vechten? Onvoorstelbaar. Dat we met Willem van Oranje wonnen van die in de waterlinies verzuipende Spanjaarden, en dat die wrede Alva het nakijken had, was helemaal mooi. Zielig dat Van Oranje dood moest. Maar ja, ons land was nu wel ontstaan.Zo ongeveer herinner ik me het. En nu zat ik zondagmiddag met mijn eigen kinderen te kijken naar de NTR-jeugdserie 'Welkom in de 80-jarige Oorlog', 450 jaar na de eerste gevechten in 1568. Ik was benieuwd welke geschiedversie zij te zien zouden krijgen.In de volwassenenversie '80 Jaar Oorlog' legt Hans Goedkoop een bom onder het beeld van goede protestanten tegen slechte katholieken uit Spanje. Het was eerder een burgeroorlog. De 'heldhaftige' geuzen waren piraten die plunderden en moordden. En de protestanten met hun beeldenstorm leken op de geloofsfanatici van IS en de Taliban, zeiden de historici.Zat ik dus kritisch te kijken hoe die visie zou doorsijpelen in de kindervariant, zag ik naast mij ogen als schoteltjes. Mijn dochter (8) en zoon (7) vergaapten zich eraan. Die kostuums! Die sfeer van opstand, intrige en macht. En\"}\n",
      "{'role': 'assistant', 'content': 'Politiek'}\n",
      "No errors found\n",
      "Num examples missing system message: 0\n",
      "Num examples missing user message: 0\n",
      "\n",
      "#### Distribution of num_messages_per_example:\n",
      "min / max: 3, 3\n",
      "mean / median: 3.0, 3.0\n",
      "p5 / p95: 3.0, 3.0\n",
      "\n",
      "#### Distribution of num_total_tokens_per_example:\n",
      "min / max: 85, 520\n",
      "mean / median: 409.02899967416096, 424.0\n",
      "p5 / p95: 328.0, 458.0\n",
      "\n",
      "#### Distribution of num_assistant_tokens_per_example:\n",
      "min / max: 2, 3\n",
      "mean / median: 2.4903877484522647, 2.0\n",
      "p5 / p95: 2.0, 3.0\n",
      "\n",
      "0 examples may be over the 4096 token limit, they will be truncated during fine-tuning\n",
      "Dataset has ~1255310 tokens that will be charged for during training\n",
      "By default, you'll train for 1 epochs on this dataset\n",
      "By default, you'll be charged for ~1255310 tokens\n"
     ]
    }
   ],
   "source": [
    "# Constants\n",
    "OPENAI_TRAIN_FILE_NAME = f'train_{MAX_WORDS}_v1'\n",
    "\n",
    "# Create and Validate OpenAI Files\n",
    "create_openai_file(train_df, f'./data/{OPENAI_TRAIN_FILE_NAME}.jsonl')\n",
    "\n",
    "# Validate Training file\n",
    "validate_openai_file(f'./data/{OPENAI_TRAIN_FILE_NAME}.jsonl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next the training file should be uploaded to OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"id\": \"file-X6xEFm1yqZq9uBi8SBJwiMQj\",\n",
      "  \"bytes\": 4462087,\n",
      "  \"created_at\": 1701804524,\n",
      "  \"filename\": \"train_192_v1.jsonl\",\n",
      "  \"object\": \"file\",\n",
      "  \"purpose\": \"fine-tune\",\n",
      "  \"status\": \"processed\",\n",
      "  \"status_details\": null\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Upload Training file to OpenAI\n",
    "ft_train_file = client.files.create(file = open(f'./data/{OPENAI_TRAIN_FILE_NAME}.jsonl', 'rb'), \n",
    "                                    purpose = 'fine-tune')\n",
    "\n",
    "# Summary\n",
    "print(ft_train_file.model_dump_json(indent = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next the validation file for OpenAI fine-tuning is created locally and validated with the code from the OpenAI Cookbook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======== ./data/validation_192_v1.jsonl\n",
      "Length Messages: 1559\n",
      "[{'messages': [{'role': 'system', 'content': 'Je bent redacteur bij een krant. Je beoordeeld een krantenartikel of het politiek of neutraal is. Hieronder staat de tekst van het krantenartikel.'}, {'role': 'user', 'content': '### Tekst:\\nVerdachte dodelijke steekpartijen Maastricht langer vastDe 37-jarige man die ervan wordt verdacht afgelopen donderdag twee mensen in Maastricht te hebben doodgestoken, blijft nog twee weken langer vastzitten. Dat heeft de rechter-commissaris vandaag beslist. De man zit in beperkingen en mag dus alleen met zijn advocaat contact hebben.Bij de steekpartijen werden een 46-jarige man bij zijn woning in Botsaartstraat gedood. Ook werd een 56-jarige vrouw in de Joseph Postmesstraat omgebracht. Daar troffen de hulpverleners nog twee gewonden aan: haar 21-jarige dochter en een 50-jarige buurtbewoner die te hulp was geschoten.Korte tijd later bleek dat geen van de aanwezigen in het Maastrichtse wijkcentrum door had dat de dader, met een bebloed gezicht en het mes nog in zijn hand, binnen was gekomen. \"We wisten nog niet wat er gebeurd was en hebben geprobeerd hem kalm te houden met een glaasje water en papiertjes om zijn gezicht schoon te vegen\\'\\', beschreef bestuurslid Osman Öztürk van cultureel centrum Tevhid.Over de toedracht van het geweld is nog niets bekend. Minister Ferdinand Grapperhaus van Justitie en Veiligheid benadrukte de volgende dag dat er geen terroristische achtergrond was. Het ging om een ernstig verwarde man.'}, {'role': 'assistant', 'content': 'Neutraal'}]}]\n",
      "\n",
      "======= OpenAI Validation\n",
      "Num examples: 1559\n",
      "First example:\n",
      "{'role': 'system', 'content': 'Je bent redacteur bij een krant. Je beoordeeld een krantenartikel of het politiek of neutraal is. Hieronder staat de tekst van het krantenartikel.'}\n",
      "{'role': 'user', 'content': '### Tekst:\\nVerdachte dodelijke steekpartijen Maastricht langer vastDe 37-jarige man die ervan wordt verdacht afgelopen donderdag twee mensen in Maastricht te hebben doodgestoken, blijft nog twee weken langer vastzitten. Dat heeft de rechter-commissaris vandaag beslist. De man zit in beperkingen en mag dus alleen met zijn advocaat contact hebben.Bij de steekpartijen werden een 46-jarige man bij zijn woning in Botsaartstraat gedood. Ook werd een 56-jarige vrouw in de Joseph Postmesstraat omgebracht. Daar troffen de hulpverleners nog twee gewonden aan: haar 21-jarige dochter en een 50-jarige buurtbewoner die te hulp was geschoten.Korte tijd later bleek dat geen van de aanwezigen in het Maastrichtse wijkcentrum door had dat de dader, met een bebloed gezicht en het mes nog in zijn hand, binnen was gekomen. \"We wisten nog niet wat er gebeurd was en hebben geprobeerd hem kalm te houden met een glaasje water en papiertjes om zijn gezicht schoon te vegen\\'\\', beschreef bestuurslid Osman Öztürk van cultureel centrum Tevhid.Over de toedracht van het geweld is nog niets bekend. Minister Ferdinand Grapperhaus van Justitie en Veiligheid benadrukte de volgende dag dat er geen terroristische achtergrond was. Het ging om een ernstig verwarde man.'}\n",
      "{'role': 'assistant', 'content': 'Neutraal'}\n",
      "No errors found\n",
      "Num examples missing system message: 0\n",
      "Num examples missing user message: 0\n",
      "\n",
      "#### Distribution of num_messages_per_example:\n",
      "min / max: 3, 3\n",
      "mean / median: 3.0, 3.0\n",
      "p5 / p95: 3.0, 3.0\n",
      "\n",
      "#### Distribution of num_total_tokens_per_example:\n",
      "min / max: 139, 507\n",
      "mean / median: 411.88069275176395, 425.0\n",
      "p5 / p95: 341.0, 461.0\n",
      "\n",
      "#### Distribution of num_assistant_tokens_per_example:\n",
      "min / max: 2, 3\n",
      "mean / median: 2.490699166132136, 2.0\n",
      "p5 / p95: 2.0, 3.0\n",
      "\n",
      "0 examples may be over the 4096 token limit, they will be truncated during fine-tuning\n",
      "Dataset has ~642122 tokens that will be charged for during training\n",
      "By default, you'll train for 1 epochs on this dataset\n",
      "By default, you'll be charged for ~642122 tokens\n"
     ]
    }
   ],
   "source": [
    "# Constants\n",
    "OPENAI_VALIDATION_FILE_NAME = f'validation_{MAX_WORDS}_v1'\n",
    "\n",
    "# Create and Validate OpenAI Files\n",
    "create_openai_file(val_df, f'./data/{OPENAI_VALIDATION_FILE_NAME}.jsonl')\n",
    "\n",
    "# Validate Validation file\n",
    "validate_openai_file(f'./data/{OPENAI_VALIDATION_FILE_NAME}.jsonl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation file should also be uploaded to OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"id\": \"file-IQPfzK3lqSpNAuy4w8cVFjHA\",\n",
      "  \"bytes\": 2278942,\n",
      "  \"created_at\": 1701804554,\n",
      "  \"filename\": \"validation_192_v1.jsonl\",\n",
      "  \"object\": \"file\",\n",
      "  \"purpose\": \"fine-tune\",\n",
      "  \"status\": \"processed\",\n",
      "  \"status_details\": null\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Upload Validation file to OpenAI\n",
    "ft_validation_file = client.files.create(file = open(f'./data/{OPENAI_VALIDATION_FILE_NAME}.jsonl', 'rb'), \n",
    "                                         purpose = 'fine-tune')\n",
    "\n",
    "# Summary\n",
    "print(ft_validation_file.model_dump_json(indent = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do a final verification to look at all the files present in OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"data\": [\n",
      "    {\n",
      "      \"id\": \"file-IQPfzK3lqSpNAuy4w8cVFjHA\",\n",
      "      \"bytes\": 2278942,\n",
      "      \"created_at\": 1701804554,\n",
      "      \"filename\": \"validation_192_v1.jsonl\",\n",
      "      \"object\": \"file\",\n",
      "      \"purpose\": \"fine-tune\",\n",
      "      \"status\": \"processed\",\n",
      "      \"status_details\": null\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"file-X6xEFm1yqZq9uBi8SBJwiMQj\",\n",
      "      \"bytes\": 4462087,\n",
      "      \"created_at\": 1701804524,\n",
      "      \"filename\": \"train_192_v1.jsonl\",\n",
      "      \"object\": \"file\",\n",
      "      \"purpose\": \"fine-tune\",\n",
      "      \"status\": \"processed\",\n",
      "      \"status_details\": null\n",
      "    }\n",
      "  ],\n",
      "  \"object\": \"list\",\n",
      "  \"has_more\": false\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# List Files\n",
    "openai_files = client.files.list()\n",
    "print(openai_files.model_dump_json(indent = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 2 files are uploaded to OpenAI and can be used in the next section to create a fine-tuning job on OpenAI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Create OpenAI fine-tuning job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we create a fine-tuning job by specifying the training and validation file Id's, the specific OpenAI model we want to finetune and the number of epochs.\n",
    "\n",
    "For the number of epochs you can leave it at the default setting of 'auto' or specify it by setting an integer value. With 'auto' OpenAI will determine the best number of epochs to use.\n",
    "\n",
    "I personally always use 1 or occassionally 2 epochs. With a good dataset that is usually more than enough to get a good quality out of a finetuned GPT-3.5 model. More epochs might lead to a small increase in model quality but the largest impact is likely only on your creditcard bill for OpenAI ;-)\n",
    "\n",
    "Batch size and learning rate multiplier I will leave at their default setting of 'auto'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"id\": \"ftjob-fHJTmhzvk4XR1V0SvDtz41bC\",\n",
      "  \"created_at\": 1701804760,\n",
      "  \"error\": null,\n",
      "  \"fine_tuned_model\": null,\n",
      "  \"finished_at\": null,\n",
      "  \"hyperparameters\": {\n",
      "    \"n_epochs\": 1,\n",
      "    \"batch_size\": \"auto\",\n",
      "    \"learning_rate_multiplier\": \"auto\"\n",
      "  },\n",
      "  \"model\": \"gpt-3.5-turbo-1106\",\n",
      "  \"object\": \"fine_tuning.job\",\n",
      "  \"organization_id\": \"org-L65zldBJfoBsfAyAAE4pGgEt\",\n",
      "  \"result_files\": [],\n",
      "  \"status\": \"validating_files\",\n",
      "  \"trained_tokens\": null,\n",
      "  \"training_file\": \"file-X6xEFm1yqZq9uBi8SBJwiMQj\",\n",
      "  \"validation_file\": \"file-IQPfzK3lqSpNAuy4w8cVFjHA\"\n",
      "}\n",
      "\n",
      "FineTuneJob ID: ftjob-fHJTmhzvk4XR1V0SvDtz41bC\n"
     ]
    }
   ],
   "source": [
    "# Create finetuned model\n",
    "fine_tune_job = client.fine_tuning.jobs.create(training_file = ft_train_file.id,\n",
    "                                               validation_file = ft_validation_file.id,\n",
    "                                               model = \"gpt-3.5-turbo-1106\",\n",
    "                                               hyperparameters = {\"n_epochs\": 1,\n",
    "                                                                  \"batch_size\": 'auto', \n",
    "                                                                  \"learning_rate_multiplier\": 'auto'})\n",
    "\n",
    "# Summary\n",
    "print(fine_tune_job.model_dump_json(indent = 2))\n",
    "\n",
    "# Get FineTuningJob Id\n",
    "ft_job_id = fine_tune_job.id\n",
    "\n",
    "# Summary\n",
    "print(f'\\nFineTuneJob ID: {ft_job_id}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating the fine-tuning job we can use its Id to retrieve the status of the fine-tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"id\": \"ftjob-fHJTmhzvk4XR1V0SvDtz41bC\",\n",
      "  \"created_at\": 1701804760,\n",
      "  \"error\": null,\n",
      "  \"fine_tuned_model\": null,\n",
      "  \"finished_at\": null,\n",
      "  \"hyperparameters\": {\n",
      "    \"n_epochs\": 1,\n",
      "    \"batch_size\": 2,\n",
      "    \"learning_rate_multiplier\": 2\n",
      "  },\n",
      "  \"model\": \"gpt-3.5-turbo-1106\",\n",
      "  \"object\": \"fine_tuning.job\",\n",
      "  \"organization_id\": \"org-L65zldBJfoBsfAyAAE4pGgEt\",\n",
      "  \"result_files\": [],\n",
      "  \"status\": \"running\",\n",
      "  \"trained_tokens\": null,\n",
      "  \"training_file\": \"file-X6xEFm1yqZq9uBi8SBJwiMQj\",\n",
      "  \"validation_file\": \"file-IQPfzK3lqSpNAuy4w8cVFjHA\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Retrieve the state of a fine-tune\n",
    "ft_job = client.fine_tuning.jobs.retrieve(ft_job_id)\n",
    "\n",
    "# Summary\n",
    "print(ft_job.model_dump_json(indent = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using the 'list_events' method we can get even more detailed information from the fine-tuning job while it is running.\n",
    "\n",
    "It will provide basic information about the progress, training and validation loss and training and validation mean token accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"data\": [\n",
      "    {\n",
      "      \"id\": \"ftevent-yJStnywH0oB9EaVDNWdjqNeP\",\n",
      "      \"created_at\": 1701808756,\n",
      "      \"level\": \"info\",\n",
      "      \"message\": \"The job has successfully completed\",\n",
      "      \"object\": \"fine_tuning.job.event\",\n",
      "      \"data\": {},\n",
      "      \"type\": \"message\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"ftevent-blnWcP6jj7oNX7rdgODUTdpH\",\n",
      "      \"created_at\": 1701808753,\n",
      "      \"level\": \"info\",\n",
      "      \"message\": \"New fine-tuned model created: ft:gpt-3.5-turbo-1106:lumi-ml-consulting::8SWUDZg1\",\n",
      "      \"object\": \"fine_tuning.job.event\",\n",
      "      \"data\": {},\n",
      "      \"type\": \"message\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"ftevent-nRsvsRpKoNPpG7bxTyz47egf\",\n",
      "      \"created_at\": 1701808677,\n",
      "      \"level\": \"info\",\n",
      "      \"message\": \"Step 1501/1535: training loss=0.00\",\n",
      "      \"object\": \"fine_tuning.job.event\",\n",
      "      \"data\": {\n",
      "        \"step\": 1501,\n",
      "        \"train_loss\": 0.0,\n",
      "        \"valid_loss\": 0.0,\n",
      "        \"train_mean_token_accuracy\": 1.0,\n",
      "        \"valid_mean_token_accuracy\": 0.0\n",
      "      },\n",
      "      \"type\": \"metrics\"\n",
      "    }\n",
      "  ],\n",
      "  \"object\": \"list\",\n",
      "  \"has_more\": true\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# List up to N events from a fine-tuning job\n",
    "list_events = client.fine_tuning.jobs.list_events(fine_tuning_job_id = ft_job_id, \n",
    "                                                  limit = 3)\n",
    "print(list_events.model_dump_json(indent = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fine-tuning takes little over an hour. After fine-tuning is finished you will receive a confirmation email.\n",
    "\n",
    "The costs for finetuning this model where 9.99 US dollars according to the OpenAI Portal Usage section. The number of tokens trained on is 1249172.\n",
    "\n",
    "If we retrieve the FineTuningJob status again we can get the identifiers for the finetuned model and the finetune metrics results file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"id\": \"ftjob-fHJTmhzvk4XR1V0SvDtz41bC\",\n",
      "  \"created_at\": 1701804760,\n",
      "  \"error\": null,\n",
      "  \"fine_tuned_model\": \"ft:gpt-3.5-turbo-1106:lumi-ml-consulting::8SWUDZg1\",\n",
      "  \"finished_at\": 1701808752,\n",
      "  \"hyperparameters\": {\n",
      "    \"n_epochs\": 1,\n",
      "    \"batch_size\": 2,\n",
      "    \"learning_rate_multiplier\": 2\n",
      "  },\n",
      "  \"model\": \"gpt-3.5-turbo-1106\",\n",
      "  \"object\": \"fine_tuning.job\",\n",
      "  \"organization_id\": \"org-L65zldBJfoBsfAyAAE4pGgEt\",\n",
      "  \"result_files\": [\n",
      "    \"file-QnwsTK2WBeXLdJfepRRRU7vA\"\n",
      "  ],\n",
      "  \"status\": \"succeeded\",\n",
      "  \"trained_tokens\": 1249172,\n",
      "  \"training_file\": \"file-X6xEFm1yqZq9uBi8SBJwiMQj\",\n",
      "  \"validation_file\": \"file-IQPfzK3lqSpNAuy4w8cVFjHA\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Retrieve the state of a fine-tune\n",
    "ft_job = client.fine_tuning.jobs.retrieve(ft_job_id)\n",
    "\n",
    "# Summary\n",
    "print(ft_job.model_dump_json(indent = 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics Result File ID: ['file-QnwsTK2WBeXLdJfepRRRU7vA']\n",
      "FineTuned Model Identifier: ft:gpt-3.5-turbo-1106:lumi-ml-consulting::8SWUDZg1\n"
     ]
    }
   ],
   "source": [
    "# Get Results File ID from FinetuningJob\n",
    "ft_file_results_id = ft_job.result_files\n",
    "print(f'Metrics Result File ID: {ft_file_results_id}')\n",
    "\n",
    "# Get FineTuned Model Identifier\n",
    "ft_model_id = ft_job.fine_tuned_model\n",
    "print(f'FineTuned Model Identifier: {ft_model_id}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The identifier of the fine-tuned model we will use in the next notebook where we will further explore the validation proces.\n",
    "\n",
    "The metrics results file is however very interresting as it will show us the detailed information of the model training and validation proces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>step</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_accuracy</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>valid_mean_token_accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>4.56670</td>\n",
       "      <td>0.55556</td>\n",
       "      <td>4.84392</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3.60616</td>\n",
       "      <td>0.60000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>6.16454</td>\n",
       "      <td>0.50000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>4.06700</td>\n",
       "      <td>0.55556</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>5.32834</td>\n",
       "      <td>0.55556</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>4.44888</td>\n",
       "      <td>0.55556</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>4.40362</td>\n",
       "      <td>0.55556</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>3.91862</td>\n",
       "      <td>0.60000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>4.42872</td>\n",
       "      <td>0.60000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>3.80956</td>\n",
       "      <td>0.60000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>4.56293</td>\n",
       "      <td>0.50000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    step  train_loss  train_accuracy  valid_loss  valid_mean_token_accuracy\n",
       "0      1     4.56670         0.55556     4.84392                        0.2\n",
       "1      2     3.60616         0.60000         NaN                        NaN\n",
       "2      3     6.16454         0.50000         NaN                        NaN\n",
       "3      4     4.06700         0.55556         NaN                        NaN\n",
       "4      5     5.32834         0.55556         NaN                        NaN\n",
       "5      6     4.44888         0.55556         NaN                        NaN\n",
       "6      7     4.40362         0.55556         NaN                        NaN\n",
       "7      8     3.91862         0.60000         NaN                        NaN\n",
       "8      9     4.42872         0.60000         NaN                        NaN\n",
       "9     10     3.80956         0.60000         NaN                        NaN\n",
       "10    11     4.56293         0.50000         NaN                        NaN"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " # Get Metric Results File\n",
    "finetune_metrics = client.files.content(ft_file_results_id[0])\n",
    "\n",
    "# Show Finetune Metrics\n",
    "metrics_df = pd.read_csv(StringIO(finetune_metrics.content.decode()))\n",
    "metrics_df.head(11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to get a quick impression of the model fine-tuning is the overview page for Fine-Tuning on the OpenAI website.\n",
    "\n",
    "![OpenAI Fine-Tuning Overview](assets/finetuning.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}