SyntheticExperts_Train_MMX_Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "68a06688",
   "metadata": {},
   "source": [
    "# Creating Synthetic Experts with Generative AI\n",
    "> ## Train MMX Synthetic Expert on AI labeled texts\n",
    "\n",
    "  \n",
    "Version 1.0  \n",
    "Date: September 2, 2023    \n",
    "Author: Daniel M. Ringel    \n",
    "Contact: dmr@unc.edu\n",
    "\n",
    "*Daniel M. Ringel, Creating Synthetic Experts with Generative Artificial Intelligence (July 15, 2023).  \n",
    "Available at SSRN: https://papers.ssrn.com/abstract_id=4542949*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "129814e4",
   "metadata": {},
   "source": [
    "**The 5,000 demo texts that this notebook uses are *Synthetic Twins* of real Tweets. I do not publish real (i.e., original) Tweets with this notebook.**\n",
    "\n",
    "> ***Synthetic Twins*** correspond semantically in idea and meaning to original texts. However, wording, people, places, firms, brands, and products were changed by an AI. As such, ***Synthetic Twins*** mitigate, to some extent, possible privacy, and copyright concerns. If you'd like to learn more about ***Synthetic Twins***, another generative AI project by Daniel Ringel, then please get in touch! dmr@unc.edu  \n",
    "\n",
    "\n",
    "You can ***create your own Synthetic Twins of texts*** with this Python notebook:   `SyntheticExperts_Create_Synthetic_Twins_of_Texts.ipynb`,   \n",
    "available as BETA version (still being tested) on the **Synthetic Experts [GitHub](https://github.com/dringel/Synthetic-Experts)** respository.<br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a043476d",
   "metadata": {},
   "source": [
    "##### Apple M1/M2 GPU MPS Requirements (Optional)\n",
    "> See Python Notebook: [Setup-MacBook-M2-Pytorch-TensorFlow-Apr2023.ipynb](https://github.com/dringel/Synthetic-Experts)  \n",
    "\n",
    "- Mac computer with Apple silicon GPU\n",
    "- macOS 12.3 or later\n",
    "- Python 3.7 or later\n",
    "- Xcode command-line tools: xcode-select --install\n",
    "\n",
    "##### If you have no GPU available, the code falls back to your CPU"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f3a0cae",
   "metadata": {},
   "source": [
    "# 1. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ac5be215",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import re\n",
    "import os\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score\n",
    "import krippendorff\n",
    "import torch\n",
    "from transformers import TrainingArguments, Trainer, EvalPrediction, AutoConfig, AutoTokenizer, AutoModelForSequenceClassification, IntervalStrategy\n",
    "from datasets import Dataset, DatasetDict\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7446ed2f",
   "metadata": {},
   "source": [
    "# 2. Configure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7acfd81a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Paths and Filenames\n",
    "IN_TrainPath = \"Data\"\n",
    "IN_TrainSample = \"Demo_5000_Labeled_SyntheticTwins\"\n",
    "Training_Path = \"Training\"\n",
    "if not os.path.exists(Training_Path): os.makedirs(Training_Path)\n",
    "\n",
    "# Set Controls\n",
    "P = 95   # percentile for max tokens\n",
    "T = 0.2  # size of test split for training\n",
    "seed = 42 # seed used everywhere\n",
    "\n",
    "# Pre-Trained LLM to fine-tune \n",
    "# ---> Select from thousands at: https://huggingface.co/models and \"plug-in\" alternative model name\n",
    "pretrained = 'roberta-large'\n",
    "\n",
    "# Set basic Hyperparameters for training (classifier performance can vary with different parameter settings)\n",
    "hyperparameters =  {'learning_rate': 6.7e-06,\n",
    "                    'per_device_train_batch_size': 16,\n",
    "                    'weight_decay': 1.1e-05,\n",
    "                    'num_train_epochs': 3,\n",
    "                    'warmup_steps': 500}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "43ba161e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorch version: 2.0.0\n",
      "GPU available! Using >>> mps <<< for training\n"
     ]
    }
   ],
   "source": [
    "print(f\"PyTorch version: {torch.__version__}\")\n",
    "device = \"mps\" if \"backends\" in dir(torch) and hasattr(torch.backends, 'mps') and torch.backends.mps.is_built() and torch.backends.mps.is_available() else \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "if device == \"cpu\": print(\"No GPU found, using >>> CPU <<< for training, which will be slow.\") \n",
    "else: print(f\"GPU available! Using >>> {device} <<< for training\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02437b64",
   "metadata": {},
   "source": [
    "# 3. Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bb523507",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tokens(text):\n",
    "    \"\"\"Tokenize text (provided tokenizer is instantiated) \"\"\"\n",
    "    return len(tokenizer(text)['input_ids'])\n",
    "\n",
    "def compute_percentile(split, P):\n",
    "    \"\"\"Compute Pth percentile of number of tokens in texts of a given split\"\"\"\n",
    "    num_tokens = [get_tokens(dataset[split][i][\"Text\"]) for i in range(len(dataset[split]))]\n",
    "    return np.percentile(num_tokens, P)\n",
    "\n",
    "def preprocess(examples, max_tokens):\n",
    "    \"\"\"Encode texts with labels for training\"\"\"\n",
    "    text = examples[\"Text\"]\n",
    "    encoding = tokenizer(text, padding=\"max_length\", truncation=True, max_length=max_tokens)\n",
    "    relevant_keys = set(examples.keys()) & set(labels)\n",
    "    labels_matrix = np.zeros((len(text), len(labels)))\n",
    "    for idx, label in enumerate(labels):\n",
    "        if label in relevant_keys:\n",
    "            labels_matrix[:, idx] = examples[label]\n",
    "    encoding[\"labels\"] = labels_matrix.tolist()\n",
    "    return encoding\n",
    "\n",
    "def multi_label_metrics(predictions: np.array, labels: np.array, threshold: float = 0.5) -> dict:\n",
    "    \"\"\"\n",
    "    Calculate classification metrics for multi-label classification.\n",
    "    :param predictions: The raw output predictions from the model.\n",
    "    :param labels: The ground truth labels.\n",
    "    :param threshold: The threshold for converting probabilities to binary predictions.\n",
    "    :return: A dictionary containing precision, recall, F1 score, ROC AUC score, and Krippendorff's alpha.\n",
    "    \"\"\"\n",
    "    sigmoid = torch.nn.Sigmoid()\n",
    "    probs = sigmoid(torch.Tensor(predictions))\n",
    "    y_pred = (probs >= threshold).numpy().astype(int)\n",
    "    av = \"micro\"\n",
    "    metrics = {\n",
    "        'precision': precision_score(y_true=labels, y_pred=y_pred, average=av),\n",
    "        'recall': recall_score(y_true=labels, y_pred=y_pred, average=av),\n",
    "        'f1': f1_score(y_true=labels, y_pred=y_pred, average=av),\n",
    "        'roc_auc': roc_auc_score(y_true=labels, y_score=probs, average=av),\n",
    "        'krippendorff_alpha': krippendorff.alpha(reliability_data=np.vstack((labels.ravel(), y_pred.ravel())))\n",
    "    }\n",
    "    return metrics\n",
    "\n",
    "def compute_metrics(eval_prediction: EvalPrediction) -> dict:\n",
    "    \"\"\"\n",
    "    Wrapper function for computing multi-label metrics using EvalPrediction object.\n",
    "    \"\"\"\n",
    "    preds = eval_prediction.predictions[0] if isinstance(eval_prediction.predictions, tuple) else eval_prediction.predictions\n",
    "    return multi_label_metrics(predictions=preds, labels=eval_prediction.label_ids)\n",
    "\n",
    "def seed_everything(seed = 42):\n",
    "    \"\"\"Seed everything for replicability. Largely works (especially on cuda, but not so much on Apple silicone (mps))\"\"\"\n",
    "    random.seed(seed)\n",
    "    os.environ['PYTHONHASHSEED'] = str(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    if device == \"cuda\":\n",
    "        torch.cuda.manual_seed(seed)\n",
    "        torch.cuda.manual_seed_all(seed)\n",
    "        torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c865df84",
   "metadata": {},
   "source": [
    "# 4. Load and Prepare Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a38cf104",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Training Data (Twins)\n",
    "TrainSample = pd.read_pickle(f\"{IN_TrainPath}/{IN_TrainSample}.pkl\")[[\"Text\", \"Product\", \"Place\", \"Price\", \"Promotion\"]].reset_index(drop=True)\n",
    "TrainSample.rename(columns={'Twin': 'Text'}, inplace=True)\n",
    "TrainSample.index.name = \"ID\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fcaa573b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the DataFrame into train and test sets, stratified by the minority label column\n",
    "minority_label = TrainSample.iloc[:, 1:].sum().idxmin()\n",
    "train, test = train_test_split(TrainSample, test_size=T, random_state=seed, stratify=TrainSample[minority_label])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "316c126b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create HuggingFaces Dataset\n",
    "dataset = DatasetDict({\"train\":Dataset.from_dict(train),\"test\":Dataset.from_dict(test)})\n",
    "\n",
    "# Get Labels and create label dicts\n",
    "labels = [label for label in dataset['train'].features.keys() if label not in ['ID', 'Text']]\n",
    "id2label = {idx:label for idx, label in enumerate(labels)}\n",
    "label2id = {label:idx for idx, label in enumerate(labels)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a1b14d09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/4000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Set Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(pretrained)\n",
    "\n",
    "# Prohibit Paralell Tokenization (can lead to forking in loops and batch processing)\n",
    "os.environ['TOKENIZERS_PARALLELISM'] = 'false'\n",
    "\n",
    "# Compute percentile for train and test splits (percentile for max tokens)\n",
    "higher_percentile = max(compute_percentile('train',P), compute_percentile('test',P))\n",
    "\n",
    "# Create encoded dataset\n",
    "encoded_dataset = dataset.map(lambda examples: preprocess(examples, int(higher_percentile)), batched=True, remove_columns=dataset['train'].column_names)\n",
    "\n",
    "# Set encoded dataset to pytorch tensors\n",
    "encoded_dataset.set_format(\"torch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30c4a5af",
   "metadata": {},
   "source": [
    "# 5. Set-up Fine-Tuning of LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "903c80fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaForSequenceClassification: ['lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.bias', 'lm_head.dense.weight', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias', 'lm_head.decoder.weight', 'lm_head.layer_norm.bias']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-large and are newly initialized: ['classifier.out_proj.weight', 'classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "# Seed Torch etc.\n",
    "seed_everything(seed)\n",
    "\n",
    "# Instantiate Classifier\n",
    "    # ---> Note: You need to set \"ignore_mismatched_sizes\" to \"True\" if fine-tuning a pre-trained classification model with different class numbers\n",
    "    # ---> You should get several warnings about weights of checkpoint not being used in initialization. \n",
    "    #      This is expected since you will train the pretrained model on downstream task.\n",
    "model = AutoModelForSequenceClassification.from_pretrained(pretrained,                                        \n",
    "                                                           problem_type=\"multi_label_classification\", \n",
    "                                                           num_labels=len(labels),\n",
    "                                                           id2label=id2label,\n",
    "                                                           label2id=label2id)\n",
    "                                                           #ignore_mismatched_sizes=True)                                                 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "daf3500e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ready to Create Synthetic Expert\n"
     ]
    }
   ],
   "source": [
    "# Set Training Arguments\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=f\"{Training_Path}\",\n",
    "    evaluation_strategy=\"epoch\",\n",
    "    logging_dir=f\"{Training_Path}/Logs\",\n",
    "    logging_strategy=\"steps\",\n",
    "    logging_steps=10,\n",
    "    per_device_train_batch_size=hyperparameters['per_device_train_batch_size'],\n",
    "    per_device_eval_batch_size= hyperparameters['per_device_train_batch_size'], \n",
    "    num_train_epochs=hyperparameters['num_train_epochs'],\n",
    "    learning_rate=hyperparameters['learning_rate'], \n",
    "    weight_decay=hyperparameters['weight_decay'], \n",
    "    warmup_steps=hyperparameters['warmup_steps'],\n",
    "    save_strategy=\"epoch\",\n",
    "    load_best_model_at_end=True,\n",
    "    save_total_limit=2,\n",
    "    use_mps_device=(device == \"mps\"),\n",
    "    optim='adamw_torch',\n",
    "    seed=seed\n",
    "    # ---> You can also do a more granular evaluation than epochs at every 100 (or so) steps\n",
    "    #evaluation_strategy=IntervalStrategy.STEPS,  # Evaluate every 'eval_steps'\n",
    "    #eval_steps=100,                              # Evaluate every 100 steps\n",
    "    #do_train=True,\n",
    "    #do_eval=True,\n",
    "    #save_strategy=IntervalStrategy.STEPS,        # Save every 'save_steps'\n",
    "    #save_steps=100,                              # Save every 100 steps\n",
    ")\n",
    "\n",
    "# Instantiate Trainer\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=encoded_dataset[\"train\"],\n",
    "    eval_dataset=encoded_dataset[\"test\"],\n",
    "    tokenizer=tokenizer,\n",
    "    compute_metrics=compute_metrics\n",
    ")\n",
    "print(\"Ready to Create Synthetic Expert\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b573c0e8",
   "metadata": {},
   "source": [
    "# 6. Fine-Tune and Evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "436b3e86",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a RobertaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started training with seed 42 at 2023-09-04 09:42:22.485229\n",
      "Fine-tuning roberta-large\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='750' max='750' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [750/750 07:02, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Roc Auc</th>\n",
       "      <th>Krippendorff Alpha</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.498500</td>\n",
       "      <td>0.491541</td>\n",
       "      <td>0.735000</td>\n",
       "      <td>0.542035</td>\n",
       "      <td>0.623939</td>\n",
       "      <td>0.826578</td>\n",
       "      <td>0.467025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.302100</td>\n",
       "      <td>0.283099</td>\n",
       "      <td>0.836991</td>\n",
       "      <td>0.787611</td>\n",
       "      <td>0.811550</td>\n",
       "      <td>0.947515</td>\n",
       "      <td>0.719186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.222400</td>\n",
       "      <td>0.264282</td>\n",
       "      <td>0.858162</td>\n",
       "      <td>0.798673</td>\n",
       "      <td>0.827349</td>\n",
       "      <td>0.953445</td>\n",
       "      <td>0.743398</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed training at 2023-09-04 09:49:25.725530\n"
     ]
    }
   ],
   "source": [
    "# Fine-tune the model with trainer to create Synthetic Expert\n",
    "print(f\"Started training with seed {seed} at {datetime.now()}\\nFine-tuning {pretrained}\")\n",
    "trainer.train()\n",
    "print(f\"Completed training at {datetime.now()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "69f7d426",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model performance on Test\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='313' max='63' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [63/63 00:38]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.26428163051605225,\n",
       " 'eval_precision': 0.8581616481774961,\n",
       " 'eval_recall': 0.7986725663716814,\n",
       " 'eval_f1': 0.8273491214667684,\n",
       " 'eval_roc_auc': 0.9534447114633678,\n",
       " 'eval_krippendorff_alpha': 0.7433975515816948,\n",
       " 'eval_runtime': 7.7485,\n",
       " 'eval_samples_per_second': 129.058,\n",
       " 'eval_steps_per_second': 8.131,\n",
       " 'epoch': 3.0}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate Synthetic Expert on test data\n",
    "print(\"Model performance on Test\")\n",
    "trainer.evaluate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2778c972",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model performance on Train\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.18405453860759735,\n",
       " 'eval_precision': 0.9181194906953967,\n",
       " 'eval_recall': 0.8694119829345205,\n",
       " 'eval_f1': 0.8931021341463415,\n",
       " 'eval_roc_auc': 0.978809225284601,\n",
       " 'eval_krippendorff_alpha': 0.8409307659295377,\n",
       " 'eval_runtime': 31.3477,\n",
       " 'eval_samples_per_second': 127.601,\n",
       " 'eval_steps_per_second': 7.975,\n",
       " 'epoch': 3.0}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  Evaluate Synthetic Expert on train data\n",
    "print(\"Model performance on Train\")\n",
    "trainer.eval_dataset = encoded_dataset[\"train\"]\n",
    "trainer.evaluate()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc477b1e",
   "metadata": {},
   "source": [
    "# 7. Save Synthetic Expert\n",
    "\n",
    "You can save your fine-tuned model and then:\n",
    "- **Load** it to classify text (e.g., use the *[SyntheticExperts_Quickstart_MMXClassifier.ipynb](https://github.com/dringel/Synthetic-Experts)* notebook)\n",
    "- **Share** it with others by sending them the model folder (consider sending them the corresponding notebooks for prediction as well)\n",
    "- **Publish** it on the Hugging Face Model Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "afd0ad16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your Synthetic Expert was saved! If you use this notebook's code, please give credit to the author by citing the paper:\n",
      "\n",
      "Daniel M. Ringel, Creating Synthetic Experts with Generative Artificial Intelligence (July 15, 2023). Available at SSRN: https://papers.ssrn.com/abstract_id=4542949\n"
     ]
    }
   ],
   "source": [
    "# Save fine-tuned model\n",
    "trainer.save_model(f\"{Training_Path}/my_MMX_SyntheticExpert_Twins\")\n",
    "print(\"Your Synthetic Expert was saved! If you use this notebook's code, please give credit to the author by citing the paper:\\n\\nDaniel M. Ringel, Creating Synthetic Experts with Generative Artificial Intelligence (July 15, 2023). Available at SSRN: https://papers.ssrn.com/abstract_id=4542949\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}