D1_04_Tokenization_and_Embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "29cd3a8e-b73f-4481-95d1-efea9317baa5",
   "metadata": {},
   "source": [
    "# Tokenization and Embeddings\n",
    "\n",
    "Before feeding text into a Transformer model, it must be converted into a numerical format that the model can understand. This process involves two key steps: **Tokenization** and **Embeddings**.\n",
    "\n",
    "### What is Tokenization?\n",
    "\n",
    "Tokenization is the process of splitting text into smaller units called tokens. Tokens can be words, subwords, or characters. Different tokenization methods are used depending on the model architecture and language.\n",
    "\n",
    "### Types of Tokenization:\n",
    "\n",
    "1. **Word Tokenization**: Splits text into individual words (e.g., \"The quick brown fox\").\n",
    "2. **Subword Tokenization**: Splits words into smaller subword units (e.g., \"Transformer\" -> \"Trans\", \"##former\").\n",
    "3. **Character Tokenization**: Splits text into individual characters (e.g., \"hello\" -> \"h\", \"e\", \"l\", \"l\", \"o\").\n",
    "\n",
    "### What are Embeddings?\n",
    "\n",
    "Embeddings are dense vector representations of tokens. They map tokens to continuous vector spaces, capturing semantic meaning and relationships. \n",
    "\n",
    "### Types of Embeddings:\n",
    "\n",
    "1. **Static Embeddings**: Pre-trained on large corpora and remain fixed during training (e.g., Word2Vec, GloVe).\n",
    "2. **Contextual Embeddings**: Generated dynamically by Transformer models based on the context of the surrounding words (e.g., BERT, GPT).\n",
    "\n",
    "Let's explore these concepts in detail and understand how they are applied in practice using the Huggingface `transformers` library.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f4357e1-1271-4360-8410-4fd21ea509c9",
   "metadata": {},
   "source": [
    "## Tokenization with Huggingface Transformers\n",
    "\n",
    "The `transformers` library provides a wide range of pre-trained tokenizers that are optimized for different Transformer models. These tokenizers convert raw text into token IDs that can be fed into a model.\n",
    "\n",
    "### Example: Tokenizing Text using a Pre-trained Tokenizer\n",
    "Let's use the BERT tokenizer to tokenize some example text.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6c1accb9-8b8a-4ecb-a7b0-2870e95f7787",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0eb60fc8-d486-4507-b728-6cb0f637178f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load a pre-trained tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b0383c84-821c-40f2-86ab-361b34b69157",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example text\n",
    "text = \"Transformers are revolutionizing NLP!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6fd1413d-81cd-41bc-bde3-b6505f72f0ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_2 = \"Simeon Harrison is so competent and handsome, he is a unicorn trainer!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0227e64b-66a8-4735-ab94-62ca28033bcb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens: ['transformers', 'are', 'revolution', '##izing', 'nl', '##p', '!']\n"
     ]
    }
   ],
   "source": [
    "# Tokenize the text\n",
    "tokens = tokenizer.tokenize(text)\n",
    "print(\"Tokens:\", tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b42e7d82-8353-486d-92ef-55fc5a2dabba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens: ['simeon', 'harrison', 'is', 'so', 'competent', 'and', 'handsome', ',', 'he', 'is', 'a', 'unicorn', 'trainer', '!']\n"
     ]
    }
   ],
   "source": [
    "tokens_2 = tokenizer.tokenize(text_2)\n",
    "print(\"Tokens:\", tokens_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "758c6590-e3c8-42e0-b366-15a8ad14fb38",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_3 = \"Large language models on supercomputers\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e69eb054-fe3f-44a0-8c7d-1ec7b88cb80b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens: ['large', 'language', 'models', 'on', 'super', '##com', '##put', '##ers']\n"
     ]
    }
   ],
   "source": [
    "tokens_3 = tokenizer.tokenize(text_3)\n",
    "print(\"Tokens:\", tokens_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f07b9d33-cabc-43e8-a987-620b3d29d92e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token IDs: [19081, 2024, 4329, 6026, 17953, 2361, 999]\n"
     ]
    }
   ],
   "source": [
    "# Let's look at the actual numeric representations of the tokens\n",
    "token_ids = tokenizer.convert_tokens_to_ids(tokens)\n",
    "print(\"Token IDs:\", token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "969fb0d5-2ce9-4a86-8b66-678f8b156693",
   "metadata": {},
   "source": [
    "## Exploring Different Tokenization Methods\n",
    "\n",
    "Huggingface provides several tokenization methods to handle various languages and tasks. Let's explore how different tokenization methods handle the same input text.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5904c76d-d4f8-4087-a0df-1e5c64561809",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Using bert-base-uncased tokenizer:\n",
      "Tokens: ['transformers', 'are', 'revolution', '##izing', 'nl', '##p', '!']\n",
      "Token IDs: [19081, 2024, 4329, 6026, 17953, 2361, 999]\n",
      "\n",
      "Using gpt2 tokenizer:\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a6c93f873fc4bda90569db0d43df52e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/26.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c24143fb441347fe9cedfc32a70bb28c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/665 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4bbf1825e64f46448bf9b7e3935ed33b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/1.04M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7da96c100cbd4c7ab630d0e8a3bad9ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "db7bd505bc9d4d5b91e4204a570eae69",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens: ['Transform', 'ers', 'Ġare', 'Ġrevolution', 'izing', 'ĠN', 'LP', '!']\n",
      "Token IDs: [41762, 364, 389, 5854, 2890, 399, 19930, 0]\n",
      "\n",
      "Using xlm-roberta-base tokenizer:\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "245b8394fa6c484eb5add32e3fa44389",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/25.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6daea2bd4aab4b678e7489873b6d2665",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/615 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "03f4b74b63be4bb2aa0a96b03403bcde",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "sentencepiece.bpe.model:   0%|          | 0.00/5.07M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9c49a543b8884bf0987e7ed0f3e57c5f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/9.10M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens: ['▁Trans', 'former', 's', '▁are', '▁revolution', 'izing', '▁N', 'LP', '!']\n",
      "Token IDs: [11062, 82772, 7, 621, 98834, 84382, 541, 37352, 38]\n"
     ]
    }
   ],
   "source": [
    "# Example of different tokenizers\n",
    "tokenizers = [\"bert-base-uncased\", \"gpt2\", \"xlm-roberta-base\"]\n",
    "\n",
    "for tok_name in tokenizers:\n",
    "    print(f\"\\nUsing {tok_name} tokenizer:\")\n",
    "    tokenizer = AutoTokenizer.from_pretrained(tok_name)\n",
    "    tokens = tokenizer.tokenize(text)\n",
    "    token_ids = tokenizer.convert_tokens_to_ids(tokens)\n",
    "    print(\"Tokens:\", tokens)\n",
    "    print(\"Token IDs:\", token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c567d830-4c26-4e52-bdef-870a044dacc0",
   "metadata": {},
   "source": [
    "## Tokenizing a Whole Dataset\n",
    "To tokenize a whole dataset, we first need to define a tokenize funktion.\n",
    "We can then use the `map()` method to apply the function to the whole dataset.\n",
    "But before that, we need to load a dataset. Let's use the emotions dataset from the Hugging Face Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b519e77b-e14e-4a42-b473-b5940edc7f36",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1563c0be-d841-430c-8b70-b8ea42233606",
   "metadata": {},
   "source": [
    "We would like to know which tokenizer is behind the variable tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f0b1badf-0e29-4988-98f3-7c9f41202593",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XLMRobertaTokenizerFast(name_or_path='xlm-roberta-base', vocab_size=250002, model_max_length=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': '<mask>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={\n",
       "\t0: AddedToken(\"<s>\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
       "\t1: AddedToken(\"<pad>\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
       "\t2: AddedToken(\"</s>\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
       "\t3: AddedToken(\"<unk>\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
       "\t250001: AddedToken(\"<mask>\", rstrip=False, lstrip=True, single_word=False, normalized=False, special=True),\n",
       "}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9cfa903-2a95-44d5-9f6b-68e6583048fd",
   "metadata": {},
   "source": [
    "As you can see, we get a lot of information about the bert-base-uncased tokenizer.\n",
    "Now, let's load the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4f4adccf-f8c4-443e-9389-a88019317a26",
   "metadata": {},
   "outputs": [],
   "source": [
    "emotions = load_dataset(\"emotion\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c353a2f5-d4fc-4b3a-abb1-db007a52d5c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 16000\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 2000\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 2000\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emotions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebbfbaac-06e3-441e-8fee-c367317d3000",
   "metadata": {},
   "source": [
    "Here, you can see the splits the dataset comes in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "52fa9904-bfca-4f50-98ee-9ba3bf9d9109",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(batch):\n",
    "    return tokenizer(batch[\"text\"], padding=True, truncation=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ddae7df-46fd-4d3b-9da8-fecbe52fbc99",
   "metadata": {},
   "source": [
    "To see `tokenize()` in action, we pass two examples from the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d19ec08b-7e3d-4390-9727-d4f3018d620b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': [[0, 17, 15935, 18, 12319, 157996, 27686, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [0, 17, 831, 738, 1295, 34698, 221, 15673, 9393, 47, 221, 121584, 297, 15673, 7844, 1660, 1295, 8035, 10932, 22008, 2750, 517, 7, 136, 83, 38075, 13, 2]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}\n"
     ]
    }
   ],
   "source": [
    "print(tokenize(emotions[\"train\"][:2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51ab7778-38d5-4573-b81f-d3fe29affafe",
   "metadata": {},
   "source": [
    "Here you can see, that a tokenized dataset does not just consist of the token ids, but much more, which the model uses for training.  \n",
    "Now, let's apply `map()`. `batch_size=None` means the tokenize function is mapped to the whole dataset as one big batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d88cbc42-19b2-4fbe-be57-3247ba686c8d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "19c1fb566dea46dcbabbe53da90b4a11",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/16000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "733f0ca60048440ba05fecca2012e25f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/2000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5d1b617a5bc469a8d8f8edf37299d75",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/2000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "emotions_encoded = emotions.map(tokenize, batched=True, batch_size=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a1315287-cec7-4c71-b229-a68ad9d69794",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['text', 'label', 'input_ids', 'attention_mask']\n"
     ]
    }
   ],
   "source": [
    "print(emotions_encoded[\"train\"].column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c67019e2-a7ea-400a-a791-17a249a81a2b",
   "metadata": {},
   "source": [
    "## Embeddings with Transformers\n",
    "\n",
    "Once text is tokenized, it is converted into embeddings before being fed into the Transformer model. Transformers then create **contextual embeddings** that capture the meaning of words in context.\n",
    "\n",
    "### Example: Extracting Embeddings from a Pre-trained Model\n",
    "Let's use a pre-trained BERT model to extract embeddings for some example text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "502a65b6-3e9d-4941-bc13-0a5c6b553d87",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModel\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "094165fd-41d1-481a-ad9c-868e94ae438e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load a pre-trained BERT model\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n",
    "model = AutoModel.from_pretrained(\"bert-base-uncased\")\n",
    "\n",
    "# Tokenize the text and convert to input IDs\n",
    "inputs = tokenizer(text, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "72ae7764-aee2-4b9c-bafd-7f1a28304a38",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101, 19081,  2024,  4329,  6026, 17953,  2361,   999,   102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1]])}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d2841471-4819-42fa-bc27-1f2b0ec243f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the embeddings (output of the model's hidden states)\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "\n",
    "# Extract the last hidden state\n",
    "last_hidden_state = outputs.last_hidden_state"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23789c3c-1264-4dff-9bb3-04813f3f4b36",
   "metadata": {},
   "source": [
    "**Explanation**  \n",
    "The `with torch.no_grad():` statement creates a context in which operations on tensors will not track gradients.\n",
    "This is used during inference (when you are using the model to make predictions rather than training it) to save memory and speed up computation by not storing information needed to compute gradients.\n",
    "\n",
    "`**inputs:` unpacks the inputs dictionary, passing its contents as keyword arguments to the model. `inputs` usually contains keys like:\n",
    "- 'input_ids': The tokenized input text.\n",
    "- 'attention_mask': A mask to differentiate real tokens from padding tokens.\n",
    "- 'token_type_ids' (optional): Indicates the segments in tasks like question answering.\n",
    "\n",
    "These inputs are then fed into the model for a forward pass.\n",
    "\n",
    "`outputs:` is a variable that will store the output of the model's forward pass. The specific structure of outputs depends on the type of model used. For most transformer models, the output will be a named tuple or an OrderedDict containing several attributes.\n",
    "\n",
    "`outputs.last_hidden_state:` extracts the \"last hidden state\" from the model's output. The last hidden state is a tensor representing the final layer's hidden states (embeddings) for each token in the input sequence.  \n",
    "The `shape` of last_hidden_state is usually `(batch_size, sequence_length, hidden_size)`, where:\n",
    "- batch_size: The number of input sequences in the batch.\n",
    "- sequence_length: The number of tokens in each input sequence.\n",
    "- hidden_size: The size of the hidden states (e.g., 768 for BERT base).  \n",
    "Purpose: The last hidden state is a contextual representation of each token in the input sequence, capturing the meaning of the token in the context of the entire sequence.\n",
    "\n",
    "Let's print that out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3d8fb0f6-2525-46a8-b1b0-9c5297e7bbd6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last Hidden State Shape: torch.Size([1, 9, 768])\n",
      "Embeddings for each token:\n",
      " tensor([[[ 0.2756,  0.0174,  0.3740,  ..., -0.1829,  0.4132,  0.2233],\n",
      "         [ 1.5966,  0.0826,  0.3611,  ..., -0.4800,  0.4948,  0.1968],\n",
      "         [ 0.5580, -0.0181,  0.1610,  ..., -0.1355,  0.6262, -0.2631],\n",
      "         ...,\n",
      "         [-0.0786, -0.6004,  1.1742,  ..., -0.1182,  0.2394, -0.1568],\n",
      "         [-0.1662, -0.4884,  0.1007,  ...,  0.9541,  0.3570, -0.4344],\n",
      "         [ 0.7650, -0.0820, -0.2475,  ...,  0.0726, -0.6258, -0.2202]]])\n"
     ]
    }
   ],
   "source": [
    "print(\"Last Hidden State Shape:\", last_hidden_state.shape)\n",
    "print(\"Embeddings for each token:\\n\", last_hidden_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "816e8024-1b83-4de3-af60-164e572f5368",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "9ef926d3-1c0b-47e9-addf-0964e1e37dad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# New text\n",
    "text = \"The fat ginger cat sat on a mat\"\n",
    "inputs = tokenizer(text, return_tensors=\"pt\")\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "\n",
    "last_hidden_state = outputs.last_hidden_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "25b343e4-ea8b-4892-9068-60880f548029",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the embeddings to numpy array\n",
    "embeddings = last_hidden_state[0].numpy()\n",
    "\n",
    "# Get the special tokens' IDs\n",
    "special_tokens_ids = tokenizer.all_special_ids\n",
    "tokens = tokenizer.convert_ids_to_tokens(inputs['input_ids'][0])\n",
    "\n",
    "# Filter out special tokens and their embeddings\n",
    "filtered_embeddings = []\n",
    "filtered_tokens = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d64dac10-71d6-4f7a-ba4e-a99082592ea1",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, token_id in enumerate(inputs['input_ids'][0]):\n",
    "    if token_id not in special_tokens_ids:\n",
    "        filtered_embeddings.append(embeddings[i])\n",
    "        filtered_tokens.append(tokens[i])\n",
    "\n",
    "# Apply PCA to reduce dimensions to 2D\n",
    "pca = PCA(n_components=2)\n",
    "reduced_embeddings = pca.fit_transform(filtered_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80039479-869d-409f-8682-b7072583cd86",
   "metadata": {},
   "source": [
    "**Explanation**  \n",
    "The `.numpy()` method converts the PyTorch tensor to a NumPy array.\n",
    "This step is necessary because PCA in scikit-learn operates on NumPy arrays, not PyTorch tensors.\n",
    "Note: For this conversion to work, the tensor must be on the CPU. If the tensor is on a GPU, you should move it to the CPU first using .cpu().  \n",
    "E.g.: `embeddings = last_hidden_state[0].cpu().numpy()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "39405aca-5dab-40b2-9156-b2631a18c6a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArEAAAIhCAYAAABZvOJuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABDzUlEQVR4nO3deVxV1f7/8fcBlUHhgKiAMqlpOGSamuWQYGWhlUVaYXWzzG90tbC6DWamlkmD17TJssGshLypWZndtIyya1ZO3fKW5hQgamoJlIYC6/cHP04ecUADNktfz8djP+KsvfY+n70lfbNYex2XMcYIAAAAsIiP0wUAAAAAx4sQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALnOSWLFmim266SfHx8apfv76aNWumAQMGaOXKlRX6JiQkyOVyyeVyycfHR0FBQTrttNM0aNAgzZkzR6WlpZV6zyFDhnjO43K55Ofnp9NPP11jx47VH3/8UaH/0qVLddVVV6lZs2aqV6+e3G63unfvrmnTpun333+v0P/AgQOKiIiQy+XSnDlzjv+mVMLq1avVu3dvud1uuVwuTZkypUKfQ6/zSNuQIUOO+X5DhgxRgwYNqv5CTsCrr7561OvJysqqsvdKSEhQ+/btq+x8RxMXF1epP4vy69+yZYunLSEhQQkJCdVWG4DjV8fpAgBUr2nTpmn37t1KS0tT27ZttXPnTv3zn//UOeecow8//FB9+vTx6t+iRQvNmjVLkvT7779r8+bNmj9/vgYNGqRevXrpvffek9vtPub7BgQEaMmSJZKkX3/9VZmZmXrooYf0ww8/aPbs2Z5+Y8eO1UMPPaTu3bvr4YcfVsuWLbV3714tW7ZM48aN0/r16/Xkk096nXvBggXasWOHJOnll1/WwIED/9I9OpybbrpJv//+u958802FhoYqLi6uQp8xY8YoNTXV83rVqlUaPny4Jk6cqMTERE9748aNq7y+mjBjxgzFx8dXaG/btq0D1Tjrueeec7oEAIcyAE5qO3bsqNBWWFhowsPDzfnnn+/V3rt3b9OuXbvDnueVV14xksxVV111zPe84YYbTP369Su09+rVy0gyubm5xhhj/vWvfxlJZujQoaa0tLRC/4KCAvPhhx9WaO/fv7+pV6+eufDCC42Pj4/Jyck5Zk3Hq06dOubWW289rmM++eQTI8m89dZbx/1+R7pnTpgxY4aRZL7++utqf6+jfc9VtdjYWHPDDTccs1/59W/evLnaawJw4phOAJzkmjRpUqGtQYMGatu2rXJycip9nhtvvFH9+vXTW2+9pZ9++umEajnnnHMkyXP8Qw89pNDQUD311FNyuVwV+gcFBalv375ebXl5efr3v/+tSy+9VHfffbdKS0v16quvVrqG7777TgMGDFBoaKj8/f3VsWNHzZw507O//FfJxcXFmjZtmudX6H/FK6+8ojPPPFP+/v5q2LChrrjiCn3//ffHPO4///mPGjVqpEsuucQzreLHH3/U4MGD1aRJE/n5+alNmzZ69tlnvY7LysqSy+VSZmamRo8eraZNmyo4OFgXXHCB1q1b95eu5VAul0sjRozQjBkzdPrppysgIEBdunTR8uXLZYzRE088oebNm6tBgwbq06ePNmzYcNjzLF26VOecc44CAgLUrFkzjRkzRiUlJV599u/frwkTJig+Pl5+fn5q3LixbrzxRu3cudOr34EDB3TPPfcoIiJCgYGB6tmzp7766qvDvu/y5cvVo0cP+fv7q2nTpho1apQOHDhQod+h0wm2bNkil8ulSZMmafLkyZ5rPPfcc7V8+fIKx7/44otq3bq1/Pz81LZtW2VkZGjIkCEVRvinTZumM888Uw0aNFBQUJDi4+N1//33H7Z24JTndIoGUPP27Nlj3G63ueKKK7zajzUq9vzzzxtJ5vXXXz/q+Y80qnjFFVcYSWb9+vUmLy/PSDJXX331cdX+yCOPGEnm/fffN6WlpSY2NtY0b978sCO5h/rhhx9MUFCQadmypXnttdfM+++/b1JSUowk89hjjxljjPn555/NF198YSSZgQMHmi+++MJ88cUXlartcCOxEydONJJMSkqKef/9981rr71mWrRoYdxut1m/fr2n36H3bPbs2cbPz8/ceuutpri42BhjzNq1a43b7TZnnHGGee2118yiRYvMXXfdZXx8fMy4ceMq1BEXF2euvfZa8/7775vMzEwTExNjWrVq5TnfkZSPRC5fvtwcOHDAazv0WEkmNjbWdO/e3cybN8+8/fbbpnXr1qZhw4bmjjvuMAMGDDALFiwws2bNMuHh4aZDhw5ef1a9e/c2YWFhpmnTpuapp54yH374obn99tuNJDN8+HBPv5KSEnPxxReb+vXrm/Hjx5vFixebl156yTRr1sy0bdvW7N271+teulwuc/fdd5tFixaZyZMnm2bNmpng4GCvkdi1a9eawMBA07ZtW5OZmWneeecdc9FFF5mYmJgKI7G9e/c2vXv39rzevHmz5x5ffPHFZv78+Wb+/PnmjDPOMKGhoWbPnj2evi+88IKRZK688krPvWjdurWJjY01sbGxnn6ZmZlGkrntttvMokWLzEcffWSef/55c/vttx/1zws4VRFigVPQtddea+rUqWNWrFjh1X6sEPvBBx94Bb4jKQ9k5cFn586dZurUqcblcpmuXbsaY4xZvny5kWTuu+++StddWlpqTjvtNNOsWTNPmBo7dqyRZD7++ONjHn/NNdcYPz8/k52d7dWelJRkAgMDvYLHoSGqMg4Nsb/++qsJCAgw/fr18+qXnZ1t/Pz8zODBgz1tB4fYRx991Pj6+la4zxdddJGJiooy+fn5Xu0jRoww/v7+5pdffvGq49D3LZ++caxQXh5iD7f5+vp69ZVkIiIizG+//eZpmz9/vpFkOnbs6BVYp0yZYiSZ//73v5623r17G0nmnXfe8TrvsGHDjI+Pj/npp5+MMX8GvLlz53r1+/rrr40k89xzzxljjPn++++NJHPHHXd49Zs1a5aR5BVir776ahMQEGC2b9/uaSsuLjbx8fGVDrFnnHGGV7D/6quvjCSTmZlpjCkL3xEREaZbt25e9fz000+mbt26XiF2xIgRJiQkxACoHKYTAKeYMWPGaNasWXryySfVuXPn4zrWGFPpvr///rvq1q2runXrqnHjxho5cqSSkpL09ttvH2/JHp9++qk2bNigG264Qb6+vpLKpjm4XC698sorxzx+yZIlOv/88xUdHe3VPmTIEO3du1dffPHFCdd2OF988YX27dtX4Yn46Oho9enTRx9//LFXuzFGt9xyi8aOHauMjAzdc889nn1//PGHPv74Y11xxRUKDAxUcXGxZ+vXr5/++OOPCr/Gvuyyy7xed+jQQZIqPR3ktdde09dff+21ffnllxX6JSYmqn79+p7Xbdq0kSQlJSV5TcUobz/0/YOCgirUOnjwYJWWluqzzz6TVPYwX0hIiC699FKva+/YsaMiIiI8KyZ88sknkqRrr73W63xXXXWV6tTxfpb5k08+0fnnn6/w8HBPm6+vr66++upj35z/r3///p7vRaniPV63bp22b9+uq666yuu4mJgY9ejRw6vt7LPP1p49e5SSkqJ33nlHu3btqnQdwKmI1QmAU8j48eM1YcIEPfLIIxoxYsRxH1/+D3PTpk2P2TcgIMATQPz8/BQbG6vg4GDP/piYGEnS5s2bK/3+L7/8siTpiiuu0J49eyRJbrdbPXv21Ny5c/XMM88oJCTkiMfv3r1bkZGRFdrLr2f37t2VrqUyys93pPdcvHixV9v+/fs1e/ZstWvXTklJSRXOVVxcrKefflpPP/30Yd/v0NATFhbm9drPz0+StG/fvkrV36ZNG3Xp0uWY/Ro2bOj1ul69ekdtP3SZtYNDZLmIiAhJf97DHTt2aM+ePZ5zHKr82sv7lx9frk6dOhXux+7duyv0O9yxR3Ose1xez+GuMTw83Ov7//rrr1dxcbFefPFFXXnllSotLVXXrl01YcIEXXjhhZWuCThVEGKBU8T48eM1btw4jRs37oQfFHn33Xflcrl03nnnHbOvj4/PUQNQZGSkzjjjDC1atEh79+5VYGDgUc+Xn5+vuXPnSpK6du162D4ZGRn6+9//fsRzhIWFadu2bRXa8/LyJEmNGjU6ag3HqzzgHOk9D30/Pz8/ffLJJ7rooot0wQUX6N///rdCQ0MlSaGhofL19dX111+v4cOHH/b9mjdvXqX115Ty5dIOtn37dkl/3sNGjRopLCxM//73vw97jqCgIK/+27dvV7NmzTz7i4uLK/yQEhYW5nmfw713VSiv52jXeLAbb7xRN954o37//Xd99tlnGjt2rC655BKtX79esbGxVVYXcDJgOgFwCnj44Yc1btw4PfDAAxo7duwJnWPGjBn64IMPlJKS4hlF/avGjBmjX3/9Vbfffvthpyr89ttvWrRokaSygLpv3z49/PDD+uSTTypsjRo1OuaUgvPPP19LlizxhNZyr732mgIDAz2rJ1SVc889VwEBAXrjjTe82nNzcz1TGw7VqVMnffrpp8rNzVVCQoJ+/vlnSVJgYKASExO1evVqdejQQV26dKmwHToqaIvCwkK9++67Xm0ZGRny8fHx/MB0ySWXaPfu3SopKTnstZ9++umS5FlBoHyt43L/+te/VFxc7NWWmJiojz/+2CtglpSUeK1j/FedfvrpioiI0L/+9S+v9uzsbC1btuyIx9WvX19JSUkaPXq09u/fr7Vr11ZZTcDJgpFY4CT3z3/+Uw8++KAuvvhi9e/fv8K8yUOD2759+zx99u3bp02bNmn+/PlasGCBevfureeff77Kahs0aJDGjBmjhx9+WD/88IOGDh3q+bCDL7/8Ui+88IKuvvpq9e3bVy+//LJCQ0P1j3/8Q/7+/hXO9be//U2TJ0/WN998ozPPPPOw7zd27FgtWLBAiYmJevDBB9WwYUPNmjVL77//vh5//PFKfYjD8QgJCdGYMWN0//33629/+5tSUlK0e/dujR8/Xv7+/kf8gaJNmzZaunSpLrjgAp133nn66KOPFBUVpalTp6pnz57q1auXbr31VsXFxamwsFAbNmzQe++95/lwiary3XffVQh+ktSyZcsq/QCHsLAw3XrrrcrOzlbr1q21cOFCvfjii7r11ls9PzBdc801mjVrlvr166e0tDSdffbZqlu3rnJzc/XJJ59owIABuuKKK9SmTRtdd911mjJliurWrasLLrhA3333nSZNmuQ1nUWSHnjgAb377rvq06ePHnzwQQUGBurZZ5897KfEnSgfHx+NHz9et9xyiwYOHKibbrpJe/bs0fjx4xUZGSkfnz/HkoYNG6aAgAD16NFDkZGR2r59u9LT0+V2u4/42wfglObwg2UAqln5099H2o7Wt379+qZFixZm4MCB5q233jIlJSWVes/jXbj/008/NQMHDjSRkZGmbt26Jjg42Jx77rnmiSeeMAUFBeabb74xkszIkSOPeI4ffvjBszzR0Xz77bfm0ksvNW6329SrV8+ceeaZZsaMGRX6qQpWJyj30ksvmQ4dOph69eoZt9ttBgwYYNauXevV53D3LDc318THx5u4uDizceNGY0zZU/E33XSTadasmalbt65p3Lix6d69u5kwYcIx6yh/ov5w13uwo61OIMm8+OKLnr6Hu0/l7/PEE08c8/6Ur4iRlZVlunTpYvz8/ExkZKS5//77zYEDB7yOP3DggJk0aZI588wzjb+/v2nQoIGJj483t9xyi/nxxx89/YqKisxdd91lmjRpYvz9/c0555xjvvjii8N+2MF//vMfc8455xg/Pz8TERFh7r77bjN9+vRKr05w6DWW35OxY8d6tU2fPt2cdtpppl69eqZ169bmlVdeMQMGDDCdOnXy9Jk5c6ZJTEw04eHhpl69eqZp06bmqquu8lrNAcCfXMYcx+PGAADgL9uzZ49at26tyy+/XNOnT3e6HMBKTCcAAKAabd++XY888ogSExMVFhamn376SU8++aQKCwuVlpbmdHmAtQixAABUIz8/P23ZskV///vf9csvv3geInz++efVrl07p8sDrMV0AgAAAFiHJbYAAABgHUIsAAAArEOIBQAAgHVOqQe7SktLlZeXp6CgILlcLqfLAQAAwCGMMSosLFTTpk29PhDkUKdUiM3Ly1N0dLTTZQAAAOAYcnJyFBUVdcT9p1SIDQoKklR2Uw79+EEAAAA4r6CgQNHR0Z7cdiSnVIgtn0IQHBxMiAUAAKjFjjX105oHu4qLi/XAAw+oefPmCggIUIsWLfTQQw+ptLTU6dIAAABQw6wZiX3sscf0/PPPa+bMmWrXrp1WrFihG2+8UW63m4/tAwAAOMVYE2K/+OILDRgwQP3795ckxcXFKTMzUytWrHC4MgAAANQ0a6YT9OzZUx9//LHWr18vSfrmm2/0+eefq1+/fkc8pqioSAUFBV4bAAAA7GfNSOy9996r/Px8xcfHy9fXVyUlJXrkkUeUkpJyxGPS09M1fvz4GqwSAAAANcGakdjZs2frjTfeUEZGhlatWqWZM2dq0qRJmjlz5hGPGTVqlPLz8z1bTk5ODVYMAACA6uIyxhini6iM6Oho3XfffRo+fLinbcKECXrjjTf0ww8/VOocBQUFcrvdys/PZ4ktAACAWqiyec2akdi9e/dW+OgxX19fltgCAAA4BVkzJ/bSSy/VI488opiYGLVr106rV6/W5MmTddNNNzldGgAAAGqYNdMJCgsLNWbMGL399tv6+eef1bRpU6WkpOjBBx9UvXr1KnUOphMAAAA4z+VyafXq1erYsWOFfZXNa9aE2KpAiAUAAHBeVYRYa+bEAgAAwDlxcXFKT09X165dVb9+fSUlJemXX37R3//+d4WEhKhVq1ZatmyZJGnWrFlq3769goKCFBMTozFjxqh83PTss8+WJHXv3l0NGjTQxIkTT6geQiwAAAAqJTMzU3PnztXWrVuVnZ2ts88+W3369NHu3bt1zTXXKDU1VZLUsGFDzZs3TwUFBXr33Xc1ffp0ZWRkSJK++uorSdKyZcv022+/6f777z+hWgixAAAAOKySEikrS8rMlP74Q0pN/btiYmIUEhKi/v37q1GjRho4cKB8fX2VkpKi7777Tvv371dSUpJat24tl8uljh07KiUlRVlZWVVamzWrEwAAAKDmzJsnpaVJubl/tj34YIQiIqTkZCkwMFARERGefYGBgTLGaO/evfrkk080fvx4rV+/XgcOHFBRUZGSkpKqtD5GYgEAAOBl3jxp4EDvACtJu3eXtc+bd+Rj9+/fr+TkZN1yyy3aunWr8vPzlZqaqoPXEnC5XH+5RkIsAAAAPEpKykZgj7Z+1ciR0pE+b6qoqEh//PGHwsLC5Ofnpy+//NIzH7ZceHi4Nm7c+JfqJMQCAADAY+nSiiOwBzNGysmRsrMPvz8oKEjPPvus/u///k/BwcF65JFHdPXVV3v1efjhh3X77bcrNDRUjz766AnVyTqxAAAA8MjMlAYPPna/jAwpJaXq3591YgEAAHDcIiOrtl91IcQCAADAo1cvKSpKOtKzVy6XFB1d1s9JhFgAAAB4+PpKU6eWfX1okC1/PWVKWT8nEWIBAADgJTlZmjNHatbMuz0qqqw9OdmZug7Ghx0AAACgguRkacCAstUKtm0rmwPbq5fzI7DlCLEAAAA4LF9fKSHB6SoOj+kEAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBAABgHUIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrWBVit27dquuuu05hYWEKDAxUx44dtXLlSqfLAgAAQA2r43QBlfXrr7+qR48eSkxM1AcffKAmTZpo48aNCgkJcbo0AAAA1DBrQuxjjz2m6OhozZgxw9MWFxfnXEEAAABwjDXTCd5991116dJFgwYNUpMmTdSpUye9+OKLRz2mqKhIBQUFXhsAAADsZ02I3bRpk6ZNm6ZWrVrpww8/VGpqqm6//Xa99tprRzwmPT1dbrfbs0VHR9dgxQAAAKguLmOMcbqIyqhXr566dOmiZcuWedpuv/12ff311/riiy8Oe0xRUZGKioo8rwsKChQdHa38/HwFBwdXe80AAAA4PgUFBXK73cfMa9aMxEZGRqpt27ZebW3atFF2dvYRj/Hz81NwcLDXBgAAAPtZE2J79OihdevWebWtX79esbGxDlUEAAAAp1gTYu+44w4tX75cEydO1IYNG5SRkaHp06dr+PDhTpcGAACAGmZNiO3atavefvttZWZmqn379nr44Yc1ZcoUXXvttU6XBgAAgBpmzYNdVaGyE4UBAADgjJPuwS4AAACgHCEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBAABgHUIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBAABgHUIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLGpEu3bttGDBAqfLAAAAJ4k6TheAU8PatWudLgEAAJxEGInFSa+4uNjpEgAAQBUjxKLK5Obm6sILL1RwcLA6d+6siRMnKi4uTpIUFxen+fPnS5JeffVVdezYUQ8//LCaNGmi8PBwTZkyxXOe0tJSPfDAAwoPD1fTpk317LPPKiQkRFlZWZ4+b775pjp06KCQkBB17dpVy5Yt8+xLSEjQPffco759+6p+/fr64IMPauDqAQBATSLE4i8pKZGysqTMTKl//8GKiYnVjh07lJmZqZdffvmIx61du1b+/v7aunWrZs+erX/84x/auHGjJGnGjBmaNWuWli5dqo0bN2rVqlUqLCz0HLtw4UL94x//0KuvvqpffvlFo0aN0qWXXqrdu3d7+rz66quaMGGCfvvtN11wwQXVdv0AAMAZhFicsHnzpLg4KTFRGjw4R//971J98MGj+uCDALVu3VqpqalHPDYsLEx333236tatq4SEBDVv3lxr1qyRJGVkZGj48OFq3bq1AgIC9Oijj6q0tNRz7LPPPqu7775bZ511lnx8fJScnKz4+HgtXLjQ02fw4ME6++yz5XK5FBAQUF23AAAAOMTaEJueni6Xy6WRI0c6Xcopad48aeBAKTe3vCVPkr+2b2+kgQPL9sfExBzx+IiICK/X9evX94y25uXlKTo62rOvcePG8vf397zesmWL7r//foWEhHi2NWvWaOvWrZ4+R3tvAABgPytXJ/j66681ffp0dejQwelSTkklJVJammTMwa1NJf0hY3bJ5WqkkSOl4cOzT+j8TZs2VU5Ojuf1zp079ccff3heR0dH67bbbjvqSK+Pj7U/nwEAgEqw7l/63377Tddee61efPFFhYaGOl3OKWnp0oNHYMtFS+oh6X4Zs085OT/q6aenn9D5U1JS9Nxzz2nDhg3at2+f7r//fq9QOmLECD3xxBNauXKljDHau3evPvroI+VWLAoAAJykrAuxw4cPV//+/Sv1sE5RUZEKCgq8Nvx127YdaU+GpE2SwiVdo3POuU5+fn7Hff6bbrpJ11xzjbp3766WLVuqY8eO8vf395zrkksu0aOPPqphw4YpNDRUzZs319SpU73mzQIAgJObyxjvXwrXZm+++aYeeeQRff311/L391dCQoI6duzotTzTwcaNG6fx48dXaM/Pz1dwcHA1V3vyysoqe5jrWIYOnagtW5boo48++kvvl5eXp2bNmiknJ0dRUVF/6VwAAKB2KygokNvtPmZes2YkNicnR2lpaXrjjTe8HvI5mlGjRik/P9+zHTzPEieuVy8pKkpyuQ7ds0rSD5KMwsNXauHCZzRo0KDjPn9xcbHmz5+vAwcO6Ndff9Udd9yhc845hwALAAA8rAmxK1eu1M8//6zOnTurTp06qlOnjj799FM99dRTqlOnjkpKSioc4+fnp+DgYK8Nf52vrzR1atnX3kF2p6QkSfVVUpKsoUOHaujQocd9fmOMHn30UYWFhalFixYqLCxURkZGFVQOAABOFtZMJygsLNRPP/3k1XbjjTcqPj5e9957r9q3b3/Mc1R2eBqVM29e2SoFBz9PFR0tTZkiJSc7VhYAALBYZfOaNUtsBQUFVQiq9evXV1hYWKUCLKpecrI0YEDZagXbtkmRkWVTDXx9na4MAACc7KwJsaidfH2lhASnqwAAAKcaq0NsVlaW0yUAAADAAdY82AUAAACUI8QCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBAABgHUIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBAABgHUIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWMeaEJuenq6uXbsqKChITZo00eWXX65169Y5XRYAAAAcYE2I/fTTTzV8+HAtX75cixcvVnFxsfr27avff//d6dIAAABQw1zGGON0ESdi586datKkiT799FOdd955lTqmoKBAbrdb+fn5Cg4OruYKAQAAcLwqm9esGYk9VH5+viSpYcOGR+xTVFSkgoICr80JO3fuVJ8+fRQcHKxBgwY5UgMAAMDJpI7TBZwIY4zuvPNO9ezZU+3btz9iv/T0dI0fP74GKzu86dOny9fXV3v27JGPz9F/bhgyZIhCQkI0ZcqUmikOAADAQlaOxI4YMUL//e9/lZmZedR+o0aNUn5+vmfLycmpoQq9bd68We3atTtmgAUAAEDlWDcn9rbbbtP8+fP12WefqXnz5sd1rBNzYgcNGqT58+fL5XKpXr16uvvuu/X5559rzZo1Ki4uVvfu3fXss88qLi5OTz31lO666y5P39jYWK1du7ZG6gQAAKgNKpvXrAmxxhjddtttevvtt5WVlaVWrVod9zmcerDr4CkCW7Zs0ffff6/ExETt379fQ4cO1Z49e7R48eIKfQEAAE41lc1r1syJHT58uDIyMvTOO+8oKChI27dvlyS53W4FBAQ4XF1FJSXS0qXStm3S9u2S213WHhcXp7i4OEmSv7+/Ro8erW7duqm0tJTpBgAAAJVkTYidNm2aJCkhIcGrfcaMGRoyZEjNF3QU8+ZJaWlSbu6fbf/5j9S7t9Sr106lpaVp6dKlnhUW9u/fr8LCQrnLky4AAACOypoQa8msB82bJw0cKB1a7m+/lbX36TNKDRrs1apVq9S4cWOtWbNGnTp18lwfo7EAAADHRmKqQiUlZSOwR8vby5YVKCAgUCEhIdq9e3eFJcDCw8O1adOmaq4UAADAboTYKrR0qfcUgkMZI+3bN15r1mxQaGioevTooaSkJK8+N998s7Zu3arQ0FB16NChmisGAACwkzWrE1SF6l6dIDNTGjz42P0yMqSUlCp/ewAAAOud9B87WxtFRlZtPwAAABweIbYK9eolRUVJLtfh97tcUnR0WT8AAACcOEJsFfL1laZOLfv60CBb/nrKlLJ+AAAAOHGE2CqWnCzNmSM1a+bdHhVV1p6c7ExdAAAAJxNr1om1SXKyNGDAn5/YFRlZNoWAEVgAAICqQYitJr6+0iEfLgYAAIAqwnQCAAAAWIcQCwAAAOsQYgEAAGAdQixOaVu2bJHL5dKePXucLgUAABwHQiwAAACsQ4jFSaOgoEAjRoxQTEyMgoOD1bVrV+Xk5Gjy5Mlq1aqVgoKC1LJlSz3zzDOeY84++2xJUlRUlBo0aKBZs2Y5VT4AADgOLLEFq5WU/Lke77RpQxQQsFfLly9XRESEvvnmGwUEBCg2NlZLlixRVFSUsrKy1K9fP3Xq1Ek9evTQV199pebNmys3N1chISFOXw4AAKgkQiysNW+elJYm5eZK0g5Jbysi4ictX95UyclSp06dJElXXnml55jExERddNFFysrKUo8ePRypGwAA/HVMJ4CV5s2TBg4sD7CS9JMkP+3YEaOBA8v2l5s1a5bOOusshYaGKiQkRAsXLtSuXbscqBoAAFQVQiysU1JSNgJrzMGtsZKKZEyOJGnkyLJ+2dnZuuGGG/T4449r586d2rNnj/r16yfz/w/28eF/AQAAbMS/4LDO0qUHj8CWC5c0QFKqjNmmnJxSvfLKauXk5MgYoyZNmsjHx0cLFy7UokWLPEc1btxYPj4+2rhxYw1eAQAA+KsIsbDOtm1H2jNTUrSkLpJC9NhjqYqNjdXo0aPVp08fhYWFafbs2brssss8RwQEBGjs2LFKSkpSSEiIMjIyqv8CAADAX+YyxvuXsiezgoICud1u5efnKzg42OlycIKysqTExGP3++QTKSGhuqsBAABVqbJ5jZFYWKdXLykqSnK5Dr/f5ZKio8v6AQCAkxMhFtbx9ZWmTi37+tAgW/56ypSyfgAA4OREiIWVkpOlOXOkZs2826OiytqTk52pCwAA1Aw+7ADWSk6WBgz48xO7IiPLphAwAgsAwMmPEAur+fry8BYAAKei455OMGTIEH322WfVUQsAAABQKccdYgsLC9W3b1+1atVKEydO1NatW6ujLgAAAOCIjjvEzp07V1u3btWIESP01ltvKS4uTklJSZozZ44OHDhQHTUCAAAAXk5odYKwsDClpaVp9erV+uqrr3Taaafp+uuvV9OmTXXHHXfoxx9/rOo6AQAAAI+/tMTWtm3btGjRIi1atEi+vr7q16+f1q5dq7Zt2+rJJ5+sqhoBAAAAL8cdYg8cOKC5c+fqkksuUWxsrN566y3dcccd2rZtm2bOnKlFixbp9ddf10MPPVQd9QIAAADHv8RWZGSkSktLlZKSoq+++kodO3as0Oeiiy5SSEhIFZQHAAAAVHTcIfbJJ5/UoEGD5O/vf8Q+oaGh2rx5818qDAAAADiS4w6x119/fXXUAQAAAFTaX3qwCwAAAHACIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxjXYh97rnn1Lx5c/n7+6tz585aunSp0yUBAACghlkVYmfPnq2RI0dq9OjRWr16tXr16qWkpCRlZ2c7XRoAAABqkMsYY5wuorK6deums846S9OmTfO0tWnTRpdffrnS09OPeXxBQYHcbrfy8/MVHBxcnaUCAADgBFQ2r1kzErt//36tXLlSffv29Wrv27evli1bdthjioqKVFBQ4LUBAADAftaE2F27dqmkpETh4eFe7eHh4dq+ffthj0lPT5fb7fZs0dHRNVEqAAAAqpk1Ibacy+Xyem2MqdBWbtSoUcrPz/dsOTk5NVEiAAAAqlkdpwuorEaNGsnX17fCqOvPP/9cYXS2nJ+fn/z8/GqiPAAAANQga0Zi69Wrp86dO2vx4sVe7YsXL1b37t0dqgoAAABOsGYkVpLuvPNOXX/99erSpYvOPfdcTZ8+XdnZ2UpNTXW6NAAAANQgq0Ls1Vdfrd27d+uhhx7Stm3b1L59ey1cuFCxsbFOlwYAAIAaZNU6sX8V68QCAADUbifdOrEAAABAOUIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBAABgHUIsAAAArEOIBQAAgHUIsQAAALAOIRZwiMvl0po1a5wuAwAAKxFigRoSFxen+fPnO10GAAAnBUIsAAAArEOIBWrAoEGDlJ2drZSUFDVo0ECpqamSpOXLl6t9+/YKDg7WZZddpvz8fM8xGzdu1KWXXqrGjRsrNjZWEyZMUGlpqVOXAABArUKIBWrAW2+9pZiYGGVmZuq3337T888/L0maPXu2Pv74Y2VnZys3N1dPPvmkJGnfvn06//zz1adPH23dulVLly7Vm2++qRkzZjh5GQAA1BqEWKAalZRIWVlSZqb0xx/SoQOp9957r8LDwxUSEqIrr7xSK1eulCQtWLBAoaGhuuOOO1SvXj3FxMQoLS1NGRkZNX8RAADUQnWcLgA4Wc2bJ6WlSbm5f7b93/+V/Tc5uey/ERERnn3169dXYWGhJGnLli367rvvFBIS4tlfWlqq6Ojo6i4bAAArEGKBajBvnjRwoGTMwa0+2r27rH3OnKMfHx0drc6dO2v58uXVWSYAANZiOgFQxUpKykZgvQOsJIVL2ihJGjny6Oe45JJLtGPHDj333HP6448/VFJSonXr1ikrK6vqCwYAwEKEWKCKLV3qPYXgT/dLekbGhCon5+9HPUeDBg300Ucf6eOPP1ZcXJzCwsI0ePBgbd++vTpKBgDAOi5jKo4XnawKCgrkdruVn5+v4OBgp8vBSSozUxo8+Nj9MjKklJTqrwcAAJtUNq8xEgtUscjIqu0HAAAqIsQCVaxXLykqSnK5Dr/f5ZKio8v6AQCAE0OIBaqYr680dWrZ14cG2fLXU6aU9QMAACeGEAtUg+TksmW0mjXzbo+KKmsvXycWAACcGNaJBapJcrI0YEDZagXbtpXNge3VixFYAACqAiEWqEa+vlJCgtNVAABw8mE6AQAAAKxDiAUAnJQmT56sVq1aKSgoSC1bttQzzzzjdEkAqhDTCQAAJ6XY2FgtWbJEUVFRysrKUr9+/dSpUyf16NHD6dIAVAFCLADgpFBS4v0g5eWXX+l5kDIxMVEXXXSRsrKyCLHASYIQCwCw3rx5UlqalJv7Z1vDhrPkdv9Tv/66WcYY7d27V82bN3euSABVijmxAACrzZsnDRzoHWClbP3yyw3avPlxTZ++U3v27FG/fv1kjHGqTABVjBALALBWSUnZCGzFbPqbJCOpie6800fvvbdQixYtqvkCAVQbK0Lsli1bNHToUDVv3lwBAQFq2bKlxo4dq/379ztdGgDAQUuXHjoCW66tpNGS+ig3N0zPPDNbl112Wc0WB6BaWTEn9ocfflBpaaleeOEFnXbaafruu+80bNgw/f7775o0aZLT5QEAHLJt29H2PvT/N2nIECklpQYKAlBjrAixF198sS6++GLP6xYtWmjdunWaNm0aIRYATmGRkVXbD4A9rAixh5Ofn6+GDRsetU9RUZGKioo8rwsKCqq7LABADerVS4qKkrZuPdy8WMnlKtvfq1fN1wagelkxJ/ZQGzdu1NNPP63U1NSj9ktPT5fb7fZs0dHRNVQhAKAm+PpKU6eWfe1yee8rfz1lijzrxQI4eTgaYseNGyeXy3XUbcWKFV7H5OXl6eKLL9agQYN08803H/X8o0aNUn5+vmfLycmpzssBADggOVmaM0dq1sy7PSqqrD052Zm6AFQvl3Fw0bxdu3Zp165dR+0TFxcnf39/SWUBNjExUd26ddOrr74qH5/jy+AFBQVyu93Kz89XcHDwCdcNAKh9Dv3Erl69GIEFbFTZvObonNhGjRqpUaNGleq7detWJSYmqnPnzpoxY8ZxB1gAwMnN11dKSHC6CgA1xYoHu/Ly8pSQkKCYmBhNmjRJO3fu9OyLiIhwsDIAAAA4wYoQu2jRIm3YsEEbNmxQVFSU1z4+QhAAAODUY8Xv5IcMGSJjzGE3AAAAnHqsCLEAAADAwQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBAABgHUIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBAABgHUIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANaxLsQWFRWpY8eOcrlcWrNmjdPlAAAAwAHWhdh77rlHTZs2dboMAAAAOMiqEPvBBx9o0aJFmjRpktOlAAAAwEF1nC6gsnbs2KFhw4Zp/vz5CgwMrNQxRUVFKioq8rwuKCiorvIAAABQg6wYiTXGaMiQIUpNTVWXLl0qfVx6errcbrdni46OrsYqAQAAUFMcDbHjxo2Ty+U66rZixQo9/fTTKigo0KhRo47r/KNGjVJ+fr5ny8nJqaYrAQAAQE1yGWOMU2++a9cu7dq166h94uLidM011+i9996Ty+XytJeUlMjX11fXXnutZs6cWan3KygokNvtVn5+voKDg/9S7QAAAKh6lc1rjobYysrOzvaaz5qXl6eLLrpIc+bMUbdu3RQVFVWp8xBiAQAAarfK5jUrHuyKiYnxet2gQQNJUsuWLSsdYAEAAHDysOLBLgAAAOBgVozEHiouLk4WzIIAAABANWEkFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBAABgHUIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsQ4gFAFSLIUOGaOTIkU6XAeAkRYgFAACAdQixAIBjmjx5smJiYhQUFKS4uDi99NJLys7O1oUXXqjGjRsrNDRU/fv315YtWyRJTz31lGbNmqXnnntODRo0ULt27Zy9AAAnHZcxxjhdRE0pKCiQ2+1Wfn6+goODnS4HAKywfv16dezYUatWrVJ8fLx27NihHTt2KDg4WN9//70SExO1f/9+DR06VHv27NHixYsllU0nCAkJ0ZQpU5y9AABWqWxeq1ODNQEALFFSIi1dKm3bJpWW+soYo7Vr1yo2Nlbh4eEKDw+XJMXFxUmS/P39NXr0aHXr1k2lpaXy8eEXfQCqF3/LAAC8zJsnxcVJiYnS4MHSdde1VGDgTI0d+4zCw8PVt29frVmzRjt37tTgwYMVHR2t4OBgnXfeedq/f78KCwudvgQApwBGYgEAHvPmSQMHSodONPv116v0669XadasfVq16kFdf/316tatm/bu3atVq1apcePGWrNmjTp16qTyWWqMxgKoTvwNAwCQVDaFIC2tYoCV1smYxTJmn+65p54CAxuoTp06KigoUGBgoEJCQrR7926NHz/e66jw8HBt2rSpxuoHcGohxAIAJJXNgc3NPdye/ZLGSApXbm6Y5s9foldffVXjx4/Xhg0bFBoaqh49eigpKcnrqJtvvllbt25VaGioOnToUANXAOBUwuoEAABJUmZm2RzYY8nIkFJSqr8eAKemyuY1RmIBAJKkyMiq7QcA1YkQCwCQJPXqJUVFSS7X4fe7XFJ0dFk/AHAaIRYAIEny9ZWmTi37+tAgW/56ypSyfgDgNEIsAMAjOVmaM0dq1sy7PSqqrD052Zm6AOBQrBMLAPCSnCwNGPDnJ3ZFRpZNIWAEFkBtQogFAFTg6yslJDhdBQAcGdMJAAAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1iHEAgAAwDqEWAAAAFiHEAsAAADrEGIBADjIjh07dNVVV6lx48aKiYnR6NGjVVxcrKysLIWEhOill15SdHS0wsLCdM899zhdLnDK4hO7AAA4yODBgxUREaHNmzdr9+7d6tevn+rXr6/u3bursLBQ3377rX788Udt3rxZXbp0Ub9+/ZTAx5sBNY6RWADAKa+kRMrKkp59dquWLFmixx//pxo0aKDY2FiNHj1ar776qiTJGKP09HT5+/urTZs26t69u1auXOlo7cCpihALADilzZsnxcVJiYnSiBG5kvx1zjkRmjevbH+LFi2Um5srSQoODlZgYKDn2Pr166uwsLDmiwbAdAIAwKlr3jxp4EDJmPKWKEl/KDd3hwYODNecOVJR0WZFRUU5WCWAwyHEAgBOSSUlUlrawQFWkppJSpT0DxnzvEaM2K2wsIm64YYbnCkSwBFZNZ3g/fffV7du3RQQEKBGjRopOTnZ6ZIAAJZaulT6/7MEDpEhaZ+kWG3b1kNnnNGfVQiAWsiakdi5c+dq2LBhmjhxovr06SNjjL799lunywIAWGrbtiPtiZA0x/Pq0kulunWlhIQE7dmzx6vn/Pnzq6k6AMdiRYgtLi5WWlqannjiCQ0dOtTTfvrppztYFQDAZpGRVdsPQM2yYjrBqlWrtHXrVvn4+KhTp06KjIxUUlKS1q5de9TjioqKVFBQ4LUBACBJvXpJUVGSy3X4/S6XFB1d1g9A7WNFiN20aZMkady4cXrggQe0YMEChYaGqnfv3vrll1+OeFx6errcbrdni46OrqmSAQC1nK+vNHVq2deHBtny11OmlPUDUPs4GmLHjRsnl8t11G3FihUqLS2VJI0ePVpXXnmlOnfurBkzZsjlcumtt9464vlHjRql/Px8z5aTk1NTlwYAsEBysjRnjtSsmXd7VFRZO88PA7WXo3NiR4wYoWuuueaofeLi4jwLSbdt29bT7ufnpxYtWig7O/uIx/r5+cnPz69qigUAnJSSk6UBA8pWK9i2rWwObK9ejMACtZ2jIbZRo0Zq1KjRMft17txZfn5+WrdunXr27ClJOnDggLZs2aLY2NjqLhMAcJLz9ZUSEpyuAsDxsGJ1guDgYKWmpmrs2LGKjo5WbGysnnjiCUnSoEGDHK4OAAAANc2KECtJTzzxhOrUqaPrr79e+/btU7du3bRkyRKFhoY6XRoAAABqmMsY7w/cO5kVFBTI7XYrPz9fwcHBTpcDAACAQ1Q2r1mxxBYAAABwMEIsAAAArEOIBQAAgHUIsQAAALAOIRYAAADWIcQCAADAOoRYAAAAWIcQCwAAAOsQYgEAAGAdQiwAAACsU8fpAmpS+SfsFhQUOFwJAAAADqc8p5XntiM5pUJsYWGhJCk6OtrhSgAAAHA0hYWFcrvdR9zvMseKuSeR0tJS5eXlKSgoSC6Xy+lyVFBQoOjoaOXk5Cg4ONjpcmo17lXlca8qj3tVOdynyuNeVR73qvJOtXtljFFhYaGaNm0qH58jz3w9pUZifXx8FBUV5XQZFQQHB58S35RVgXtVedyryuNeVQ73qfK4V5XHvaq8U+leHW0EthwPdgEAAMA6hFgAAABYhxDrID8/P40dO1Z+fn5Ol1Lrca8qj3tVedyryuE+VR73qvK4V5XHvTq8U+rBLgAAAJwcGIkFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hNha5P3331e3bt0UEBCgRo0aKTk52emSarWioiJ17NhRLpdLa9ascbqcWmfLli0aOnSomjdvroCAALVs2VJjx47V/v37nS6tVnjuuefUvHlz+fv7q3Pnzlq6dKnTJdU66enp6tq1q4KCgtSkSRNdfvnlWrdundNl1Xrp6elyuVwaOXKk06XUWlu3btV1112nsLAwBQYGqmPHjlq5cqXTZdUqxcXFeuCBBzx/h7do0UIPPfSQSktLnS6t1jilPrGrNps7d66GDRumiRMnqk+fPjLG6Ntvv3W6rFrtnnvuUdOmTfXNN984XUqt9MMPP6i0tFQvvPCCTjvtNH333XcaNmyYfv/9d02aNMnp8hw1e/ZsjRw5Us8995x69OihF154QUlJSfrf//6nmJgYp8urNT799FMNHz5cXbt2VXFxsUaPHq2+ffvqf//7n+rXr+90ebXS119/renTp6tDhw5Ol1Jr/frrr+rRo4cSExP1wQcfqEmTJtq4caNCQkKcLq1Weeyxx/T8889r5syZateunVasWKEbb7xRbrdbaWlpTpdXOxg47sCBA6ZZs2bmpZdecroUayxcuNDEx8ebtWvXGklm9erVTpdkhccff9w0b97c6TIcd/bZZ5vU1FSvtvj4eHPfffc5VJEdfv75ZyPJfPrpp06XUisVFhaaVq1amcWLF5vevXubtLQ0p0uqle69917Ts2dPp8uo9fr3729uuukmr7bk5GRz3XXXOVRR7cN0glpg1apV2rp1q3x8fNSpUydFRkYqKSlJa9eudbq0WmnHjh0aNmyYXn/9dQUGBjpdjlXy8/PVsGFDp8tw1P79+7Vy5Ur17dvXq71v375atmyZQ1XZIT8/X5JO+e+hIxk+fLj69++vCy64wOlSarV3331XXbp00aBBg9SkSRN16tRJL774otNl1To9e/bUxx9/rPXr10uSvvnmG33++efq16+fw5XVHoTYWmDTpk2SpHHjxumBBx7QggULFBoaqt69e+uXX35xuLraxRijIUOGKDU1VV26dHG6HKts3LhRTz/9tFJTU50uxVG7du1SSUmJwsPDvdrDw8O1fft2h6qq/YwxuvPOO9WzZ0+1b9/e6XJqnTfffFOrVq1Senq606XUeps2bdK0adPUqlUrffjhh0pNTdXtt9+u1157zenSapV7771XKSkpio+PV926ddWpUyeNHDlSKSkpTpdWaxBiq9G4cePkcrmOuq1YscIzSXv06NG68sor1blzZ82YMUMul0tvvfWWw1dRMyp7r55++mkVFBRo1KhRTpfsmMreq4Pl5eXp4osv1qBBg3TzzTc7VHnt4nK5vF4bYyq04U8jRozQf//7X2VmZjpdSq2Tk5OjtLQ0vfHGG/L393e6nFqvtLRUZ511liZOnKhOnTrplltu0bBhwzRt2jSnS6tVZs+erTfeeEMZGRlatWqVZs6cqUmTJmnmzJlOl1Zr8GBXNRoxYoSuueaao/aJi4tTYWGhJKlt27aedj8/P7Vo0ULZ2dnVWmNtUdl7NWHCBC1fvrzC50d36dJF11577SnxP3dl71W5vLw8JSYm6txzz9X06dOrubrar1GjRvL19a0w6vrzzz9XGJ1Fmdtuu03vvvuuPvvsM0VFRTldTq2zcuVK/fzzz+rcubOnraSkRJ999pmeeeYZFRUVydfX18EKa5fIyEivf+8kqU2bNpo7d65DFdVOd999t+677z7P3/dnnHGGfvrpJ6Wnp+uGG25wuLragRBbjRo1aqRGjRods1/nzp3l5+endevWqWfPnpKkAwcOaMuWLYqNja3uMmuFyt6rp556ShMmTPC8zsvL00UXXaTZs2erW7du1VlirVHZeyWVLWOTmJjoGd338eGXL/Xq1VPnzp21ePFiXXHFFZ72xYsXa8CAAQ5WVvsYY3Tbbbfp7bffVlZWlpo3b+50SbXS+eefX2E1mRtvvFHx8fG69957CbCH6NGjR4Wl2tavX3/K/HtXWXv37q3wd7avry9LbB2EEFsLBAcHKzU1VWPHjlV0dLRiY2P1xBNPSJIGDRrkcHW1y6HLHzVo0ECS1LJlS0aIDpGXl6eEhATFxMRo0qRJ2rlzp2dfRESEg5U5784779T111+vLl26eEaos7OzT/n5wocaPny4MjIy9M477ygoKMgzeu12uxUQEOBwdbVHUFBQhXnC9evXV1hYGPOHD+OOO+5Q9+7dNXHiRF111VX66quvNH36dH5TdIhLL71UjzzyiGJiYtSuXTutXr1akydP1k033eR0abWHs4sjoNz+/fvNXXfdZZo0aWKCgoLMBRdcYL777juny6r1Nm/ezBJbRzBjxgwj6bAbjHn22WdNbGysqVevnjnrrLNYNuowjvT9M2PGDKdLq/VYYuvo3nvvPdO+fXvj5+dn4uPjzfTp050uqdYpKCgwaWlpJiYmxvj7+5sWLVqY0aNHm6KiIqdLqzVcxhjjSHoGAAAAThAT5AAAAGAdQiwAAACsQ4gFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAYJmdO3cqIiJCEydO9LR9+eWXqlevnhYtWuRgZQBQc1zGGON0EQCA47Nw4UJdfvnlWrZsmeLj49WpUyf1799fU6ZMcbo0AKgRhFgAsNTw4cP10UcfqWvXrvrmm2/09ddfy9/f3+myAKBGEGIBwFL79u1T+/btlZOToxUrVqhDhw5OlwQANYY5sQBgqU2bNikvL0+lpaX66aefnC4HAGoUI7EAYKH9+/fr7LPPVseOHRUfH6/Jkyfr22+/VXh4uNOlAUCNIMQCgIXuvvtuzZkzR998840aNGigxMREBQUFacGCBU6XBgA1gukEAGCZrKwsTZkyRa+//rqCg4Pl4+Oj119/XZ9//rmmTZvmdHkAUCMYiQUAAIB1GIkFAACAdQixAAAAsA4hFgAAANYhxAIAAMA6hFgAAABYhxALAAAA6xBiAQAAYB1CLAAAAKxDiAUAAIB1CLEAAACwDiEWAAAA1vl/1bGr73kHNDwAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 800x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the embeddings\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.scatter(reduced_embeddings[:, 0], reduced_embeddings[:, 1], c='blue')\n",
    "\n",
    "# Annotate the points with tokens\n",
    "for i, token in enumerate(filtered_tokens):\n",
    "    plt.text(reduced_embeddings[i, 0], reduced_embeddings[i, 1], token, fontsize=9)\n",
    "\n",
    "plt.title(\"2D PCA of Token Embeddings\")\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"y\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e3f73a2-cdb8-4f2c-aa64-4e35806c9d4a",
   "metadata": {},
   "source": [
    "## Hands-On Exercise: Working with Tokenization and Embeddings\n",
    "\n",
    "In this exercise, you will:\n",
    "1. Choose a different pre-trained model from the Huggingface Hub.\n",
    "2. Tokenize a new text input.\n",
    "3. Extract and visualize embeddings for the tokens.\n",
    "\n",
    "### Instructions:\n",
    "\n",
    "1. Choose a model (e.g., \"distilbert-base-uncased\").\n",
    "2. Tokenize the following text: \"Huggingface makes NLP easy and accessible.\"\n",
    "3. Extract embeddings using the last hidden state of the model.\n",
    "\n",
    "Try it out below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "78ff5979-4d7c-4222-9fd2-f4ccf8a29d1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49f89739-7b86-4d62-ada4-a56a18d7d436",
   "metadata": {},
   "source": [
    "### (Possible) Solution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1cd0b1c4-8f0d-4ce0-9042-44bcc8feb499",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0ae5156d74a478b824f5f5e83564dca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/48.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d3329b3c4d4c4f8baaeff2193e2698c0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/483 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b07b1260a7c41258ed92c75c3c0863e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f75fd6d99a074d01b52183864be41541",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0e9db703db6d4da2b814a27174925a6e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/268M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Step 1: Choose a model\n",
    "model_name = \"distilbert-base-uncased\"  # Change this to any model of your choice\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "model = AutoModel.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3cd8e5a1-c59a-4eef-bba0-07e32ade7520",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 2: Tokenize the new text\n",
    "new_text = \"Huggingface makes NLP easy and accessible.\"\n",
    "inputs = tokenizer(new_text, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "1fa25a6b-077f-415f-903b-955dfea850e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3: Extract embeddings\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "\n",
    "# Extract the last hidden state\n",
    "new_last_hidden_state = outputs.last_hidden_state"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ef312c3-199d-4480-8146-c1d5113b97a4",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this notebook, we explored the concepts of tokenization and embeddings, which are crucial for preparing text input for Transformer models. We demonstrated how to tokenize text using different methods and extract contextual embeddings from pre-trained models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "e7df86d3-451a-48b8-913d-4ae00ccaef58",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'status': 'ok', 'restart': False}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shut down the kernel to release memory\n",
    "import IPython\n",
    "\n",
    "app = IPython.Application.instance()\n",
    "app.kernel.do_shutdown(restart=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "980eab88-fed3-4543-957b-9360535c8bc5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c516da7-61b8-42ba-a916-af162db41b42",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}