HW_2_Latent_Semantic_and_skip_gram.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Submission Deadline: __October 20, 2023; 11:59 PM__\n",
        "\n",
        "A penalty will be applied for late submission. Please refer to the course policy for more detail.  \n",
        "\n",
        "## Instructions\n",
        "\n",
        "Please read the instructions carefully before you start working on the homework.\n",
        "\n",
        "- Please follow instructions and printed out the results as required. Keep the printed results and your implementation for grading purpose.\n",
        "    - The TAs will not run your code for grading purpose unless it is necessary. That means, you may lose some points if the printed results are not in the submitted file.\n",
        "- Submission should be via Canvas.\n",
        "    - If you use Google Colab for running the code, please download the file and submit it via Canvas once it's done.\n",
        "    - Submission via a Google Colab link will be considered as an invalid submission.\n",
        "- Please double check the submitted file once you upload it to Canvas.\n",
        "    - Students should be responsible for checking whether they submit the right files.\n",
        "    - Re-submission is not allowed once the deadline is passed.\n",
        "\n",
        "Also, if you missed the class lectures, please study the course materials first before working on the homework. It may save you some time."
      ],
      "metadata": {
        "id": "8joq98nXQwdi"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mbi1ud-6beud"
      },
      "source": [
        "# Homework 02 Word Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kga3AJlUbeuf"
      },
      "source": [
        "### Goal\n",
        "\n",
        "The **goal** of this homework is to provide an opportunity to build an end-to-end system.\n",
        "\n",
        "Specifically, we are going to build a word embedding system, that can\n",
        "\n",
        "1. Read and preprocess raw data\n",
        "2. Use two different ways (latent semantic analysis and skip-gram) to learn word embeddings\n",
        "3. Evaluate the quality of word embeddings using some intrinsic evaluation methods\n",
        "\n",
        "### Submission\n",
        "\n",
        "Your submission should only include this notebook file. Please keep **all the outputs** in your submission for grading. We will run the code only if we are not sure it is correct.\n",
        "\n",
        "### Dependency\n",
        "\n",
        "You will need the following package to finish this homework assignment\n",
        "\n",
        "- [spaCy](https://pypi.org/project/spacy/)\n",
        "- [fasttext](https://pypi.org/project/fasttext/)\n",
        "\n",
        "### Hint\n",
        "\n",
        "Search for the keyword `TODO` to find out which parts need your input"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "foccJkL4beug",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4c159e29-a4e6-4159-df69-c3807283e6c9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading ...\n",
            "Decompressing the file ...\n",
            "Archive:  embeddings.zip\n",
            "   creating: embeddings/\n",
            "  inflating: embeddings/imdb-small.txt  \n",
            "  inflating: embeddings/word-pairs.txt  \n",
            "Read 10000 sentences\n"
          ]
        }
      ],
      "source": [
        "# Download the data from course webpage\n",
        "import urllib.request\n",
        "from os.path import isfile\n",
        "if not isfile(\"embeddings/imdb-small.txt\"):\n",
        "    url = \"https://yangfengji.net/uva-nlp-grad/data/embeddings.zip\"\n",
        "    print(\"Downloading ...\")\n",
        "    filename, headers = urllib.request.urlretrieve(url, filename=\"embeddings.zip\")\n",
        "\n",
        "    print(\"Decompressing the file ...\")\n",
        "    !unzip embeddings.zip\n",
        "\n",
        "sents = open(\"embeddings/imdb-small.txt\").read().split(\"\\n\")\n",
        "print(\"Read {} sentences\".format(len(sents)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0C2HysKMbeuh"
      },
      "source": [
        "## 1. Data Processing (5 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gv9x86d-beuh"
      },
      "source": [
        "Data processing is an **essential** skill for NLP researchers. Unlike machine learning where researchers sometimes may want to use synthetic data to demonstrate the potential of their algorithms, NLP researchers need to deal with real-world data all the time. Unfortunately, this means that these data are noisy and often contain irregular patterns. Therefore, a reasonable data processing can alleviate the challenge of building NLP systems to some extent and may also help boost the performance of machine learning models.\n",
        "\n",
        "Data processing for learning word embeddings includes two basic modules\n",
        "\n",
        "- Tokenizing texts and replacing some special tokens\n",
        "- Filtering low-frequency and building a vocab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YqAnj84Tbeuh"
      },
      "source": [
        "### 1.1 Tokenization (2 points)\n",
        "\n",
        "The following function *tokenize()* should include the following components\n",
        "\n",
        "1. Load the raw text from the file named **imdb-small.txt**\n",
        "2. Convert all characters into lower cases\n",
        "3. Tokenize the raw text using `nltk.tokenize`\n",
        "4. Remove all punctuation (as single tokens) and replace all numbers with a special token `<num>`\n",
        "5. Write the preprocessed text to the file named **imdb-small.txt.tokenized** and maintain the same format (one paragraph per line)\n",
        "\n",
        "(The file names are pre-defined, please do not change them.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "NQ9dtItlbeuh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d23c0ade-d6ac-4d10-e438-0816723aa695"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        }
      ],
      "source": [
        "# TODO: add necessary packages here\n",
        "#import nltk\n",
        "from nltk.tokenize import RegexpTokenizer\n",
        "import string\n",
        "import re\n",
        "nltk.download('punkt')\n",
        "def tokenize(infname=\"embeddings/imdb-small.txt\"):\n",
        "    outfname = open(infname + \".tokenized\", \"w\")\n",
        "\n",
        "    with open(infname, 'r', encoding='utf-8') as file:\n",
        "        raw_text = file.read()\n",
        "    raw_text = raw_text.lower()\n",
        "\n",
        "    sentences = nltk.sent_tokenize(raw_text)\n",
        "\n",
        "    def replace_numbers(text):\n",
        "        return re.sub(r'\\d+', '<num>', text)\n",
        "\n",
        "    tokenized_sentences = []\n",
        "    tokenizer = RegexpTokenizer(r'\\w+')\n",
        "    for sentence in sentences:\n",
        "        tokens=tokenizer.tokenize(sentence)\n",
        "        cleaned_tokens = [replace_numbers(token) for token in tokens]\n",
        "        tokenized_sentences.append(' '.join(cleaned_tokens))\n",
        "\n",
        "    # Write the preprocessed text to the output file\n",
        "\n",
        "    outfname.write('\\n'.join(tokenized_sentences))\n",
        "\n",
        "\n",
        "    # ----------------------------------------\n",
        "    # TODO: add your code here\n",
        "\n",
        "    # ----------------------------------------\n",
        "    outfname.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o-yYsfWmbeui"
      },
      "source": [
        "### 1.2 Filtering (2 points)\n",
        "\n",
        "The following function *token_filter()* should include the following components\n",
        "\n",
        "1. Remove the words that appear in the data less than 5 times (word_frequency < 5)\n",
        "2. Write the filtered data to the file named **imdb-small.txt.filtered** and maintain the same format (one sentence per line)\n",
        "3. Return a Python list that contains all the words"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "bcWBcJoSbeui"
      },
      "outputs": [],
      "source": [
        "# TODO: add necessary packages here\n",
        "import nltk\n",
        "from nltk.tokenize import MWETokenizer\n",
        "\n",
        "def token_filter(infname=\"embeddings/imdb-small.txt.tokenized\", thresh=5):\n",
        "    outfname = open(infname.replace(\".tokenized\", \".filtered\"), 'w')\n",
        "    vocab = []\n",
        "    with open(infname, 'r', encoding='utf-8') as file:\n",
        "        sentences = file.readlines()\n",
        "\n",
        "    #sentences = nltk.sent_tokenize(raw_text)\n",
        "    freq_dist = nltk.FreqDist()\n",
        "\n",
        "    def custom_word_tokenize(text):\n",
        "    # Use regular expressions to tokenize words\n",
        "      words = re.findall(r'\\b\\w+\\b|<num>', text)\n",
        "      return words\n",
        "\n",
        "    for s in sentences:\n",
        "      tokens=custom_word_tokenize(s)\n",
        "      freq_dist.update(tokens)\n",
        "\n",
        "    filtered_words = [word for word in freq_dist if freq_dist[word] >= thresh]\n",
        "\n",
        "    filtered_words_set = set(filtered_words)\n",
        "    tokenized_sentences = []\n",
        "    for sentence in sentences:\n",
        "        tokens = custom_word_tokenize(sentence)\n",
        "        # Filter tokens based on the frequency\n",
        "        filtered_tokens = [token for token in tokens if token in filtered_words_set]\n",
        "        if filtered_tokens:\n",
        "            #outfname.write(' '.join(filtered_tokens) + '\\n')\n",
        "            tokenized_sentences.append(' '.join(filtered_tokens))\n",
        "\n",
        "\n",
        "    vocab=list(filtered_words_set)\n",
        "    outfname.write('\\n'.join(tokenized_sentences))\n",
        "\n",
        "    # ----------------------------------------\n",
        "    # TODO: remove \"pass\" and add your code here\n",
        "\n",
        "    # ----------------------------------------\n",
        "    outfname.close()\n",
        "    return vocab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1CZox3VFbeui"
      },
      "source": [
        "### 1.3 Put all together (1 point)\n",
        "\n",
        "The following code block will call the previous two functions to do data preprocessing.\n",
        "\n",
        "This code block should include the following steps\n",
        "\n",
        "- tokenization\n",
        "- build the vocabulary with the variable name `vocab`\n",
        "- print out the size of the vocabulary"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "syFxe2uobeui",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9018a461-6dec-4cc8-c3c4-b6c27617466e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The vocab size = 18289\n"
          ]
        }
      ],
      "source": [
        "tokenize()\n",
        "vocab = token_filter()\n",
        "print(\"The vocab size = {}\".format(len(vocab)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YMVhCnpsbeui"
      },
      "source": [
        "## 2. Word Embeddings (5 points)\n",
        "\n",
        "In this section, you need to implement two different ways of constructing word embeddings: latent semantic analysis  and skipgram."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ptLnV4mHbeuj"
      },
      "source": [
        "### 2.1 Latent semantic analysis (3 points)\n",
        "\n",
        "The function of LSA should include the following components\n",
        "\n",
        "- Construct the word-doc matrix using `CountVectorizer` with `tokenizer=lambda x : x.split()`, make sure in this matrix that each row represents one word and each column represents one document (sentence, to be accurate in this case)\n",
        "- Use the `TruncatedSVD` from `sklearn.decomposition` to factorize the word-doc matrix\n",
        "- Construct the word embedding matrix with dimensionality as $v \\times k$, where $v$ is the vocab size and $k$ is the word embedding dimension\n",
        "\n",
        "The LSA() function should return\n",
        "\n",
        "- **embeddings**: A matrix with size $v\\times k$ that contains all the word embeddings\n",
        "- **vocab**: A Python dict with size $v$ that maps a word to the corresponding word index. Please pay attention to the mapping relation in vocab, which will be needed in the evaluation section."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "Suai-OVWP5PG"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "5zI4dJnFbeuj"
      },
      "outputs": [],
      "source": [
        "# TODO: add necessary packages here\n",
        "from sklearn.decomposition import TruncatedSVD\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "import numpy as np\n",
        "def LSA(fname = \"embeddings/imdb-small.txt.filtered\", dim=50):\n",
        "    sents = open(fname).read().split(\"\\n\")\n",
        "    #print(\"stupid\")\n",
        "\n",
        "\n",
        "    # Create a CountVectorizer and transform the sentences into a word-doc matrix\n",
        "    vectorizer = CountVectorizer(tokenizer=lambda x: x.split())\n",
        "    word_doc_matrix = vectorizer.fit_transform(sents)\n",
        "    word_doc_matrix=word_doc_matrix.T\n",
        "\n",
        "    svd=TruncatedSVD(n_components=dim)\n",
        "    embeddings=svd.fit_transform(word_doc_matrix)\n",
        "    #print(embeddings.shape)\n",
        "    vocab = {word: idx for idx, word in enumerate(vectorizer.get_feature_names_out())}\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "    # -------------------------------------\n",
        "    # TODO: add your code here\n",
        "\n",
        "    # -------------------------------------\n",
        "    return embeddings, vocab"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "em,vocab=LSA()\n",
        "print(em.shape)\n",
        "print(len(vocab))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y9rSLB5zRcWY",
        "outputId": "caed90ad-aade-4825-8986-753b4e04cd8c"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/feature_extraction/text.py:528: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(18289, 50)\n",
            "18289\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v17bhMzSbeuj"
      },
      "source": [
        "### 2.2 Skip-gram model (2 points)\n",
        "\n",
        "In this section, you do not have to implement the skip-gram model by yourself. An authentic implementation of skip-gram can be found in the Python package [fasttext](https://pypi.org/project/fasttext/), which you can install on the your local machine with the folllwing commandline or directly load the package if you are using Google Colab.\n",
        "```python\n",
        "pip install fasttext\n",
        "```\n",
        "\n",
        "In the following code, please use the `fasttext.train_unsupervised` function for the skipgram() implementation. For the `fasttext.train_unsupervised`, please use the following configurations\n",
        "\n",
        "- `model='skipgram'`\n",
        "- Context window size: `ws = 3`\n",
        "- Word embedding dimension: `dim = 50`\n",
        "- Number of negative examples: `neg = 5`\n",
        "\n",
        "For all other parameters, use their default values.\n",
        "\n",
        "Similar to the previous LSA(), Skipgram() should return\n",
        "\n",
        "- **embeddings**: A matrix with size $v\\times k$ that contains all the word embeddings\n",
        "- **vocab**: A Python dict with size $v$ that maps an index to the corresponding word\n",
        "\n",
        "To get the word embeddings and vocab from fasttext, you need to understand [some functions](https://pypi.org/project/fasttext/#api) provided by the `model` object in the fasttext."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install fasttext"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J_cgYrybVK1D",
        "outputId": "e7f65426-9e0b-4b80-8ade-402fd59df482"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting fasttext\n",
            "  Downloading fasttext-0.9.2.tar.gz (68 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/68.8 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m68.8/68.8 kB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting pybind11>=2.2 (from fasttext)\n",
            "  Using cached pybind11-2.11.1-py3-none-any.whl (227 kB)\n",
            "Requirement already satisfied: setuptools>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from fasttext) (67.7.2)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from fasttext) (1.23.5)\n",
            "Building wheels for collected packages: fasttext\n",
            "  Building wheel for fasttext (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fasttext: filename=fasttext-0.9.2-cp310-cp310-linux_x86_64.whl size=4199769 sha256=d690c8ba8c429f5420fe76a1442cb53def9552ccb7f868601fc32e5d25272333\n",
            "  Stored in directory: /root/.cache/pip/wheels/a5/13/75/f811c84a8ab36eedbaef977a6a58a98990e8e0f1967f98f394\n",
            "Successfully built fasttext\n",
            "Installing collected packages: pybind11, fasttext\n",
            "Successfully installed fasttext-0.9.2 pybind11-2.11.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "cQ0arCTEbeuj"
      },
      "outputs": [],
      "source": [
        "# TODO: add necessary packages here\n",
        "import fasttext\n",
        "def Skipgram(fname = \"embeddings/imdb-small.txt.filtered\", ws=3, dim=50):\n",
        "    # ------------------------------------------\n",
        "    # TODO: add your code here\n",
        "    model = fasttext.train_unsupervised(fname, model='skipgram', ws=ws, dim=dim, neg=5)\n",
        "\n",
        "    embeddings = np.array([model.get_word_vector(word) for word in model.words])\n",
        "    vocab = {word: idx for idx, word in enumerate(model.words)}\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "    # ------------------------------------------\n",
        "    return embeddings, vocab"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "embeddings_sg, vocab_sg = Skipgram()\n",
        "print(embeddings_sg.shape)\n",
        "print(len(vocab_sg))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uOgoUv4hWfxi",
        "outputId": "d4d85d0b-afc9-4fd2-ff30-74b47d49552b"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(18290, 50)\n",
            "18290\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PPy2ImAlbeuj"
      },
      "source": [
        "### 2.3 Put all together\n",
        "\n",
        "Run the following code blocks to get word embeddings from two different methods. It may take a couple of minutes to compute both embeddings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "Ahi-Z01zbeuj"
      },
      "outputs": [],
      "source": [
        "embeddings_lsa, vocab_lsa = LSA()\n",
        "embeddings_sg, vocab_sg = Skipgram()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eKbl52S2beuj"
      },
      "source": [
        "The following code will serve as the sanity check that `vocab_lsa` and `vocab_sg` contain the same words"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "abTUwVVRbeuj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "62bdd7ba-7f69-425a-f53d-b9b1716c98af"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The word that only appear in one vocab: {'</s>'}\n"
          ]
        }
      ],
      "source": [
        "lsa_word_set = set([item[0] for item in vocab_lsa.items()])\n",
        "sg_word_set = set([item[0] for item in vocab_sg.items()])\n",
        "sym_diff = lsa_word_set.symmetric_difference(sg_word_set)\n",
        "\n",
        "if len(sym_diff) == 0:\n",
        "    print(\"vocab_lsa and vocab_sg contain the same words!\")\n",
        "else:\n",
        "    print(\"The word that only appear in one vocab: {}\".format(sym_diff))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lyl_XqYlbeuj"
      },
      "source": [
        "If the only word from the `symmetric_difference()` function is `</s>`, then your implementation should be fine. (`</s>` was added by `fasttext` automatically to the end of each text.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3av8qPGMbeuj"
      },
      "source": [
        "## 3. Evaluation (5 points)\n",
        "\n",
        "In this homework, we will only use intrinsic evaluation. Specifically, for a list of predefined word pairs with their similarity scores, the evaluation is to calculate the correlation between the predefined similarity scores and the cosine similarity scores based on word embeddings. The higher the correlation, the better the quality of word embeddings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "xLlF2ZcBbeuk"
      },
      "outputs": [],
      "source": [
        "def load_wordpairs(fname = \"embeddings/word-pairs.txt\", vocab=vocab):\n",
        "    records = {}\n",
        "    with open(fname) as fin:\n",
        "        for line in fin:\n",
        "            items = line.strip().split(\",\")\n",
        "            if (items[1] in vocab) and (items[2] in vocab): # make sure both words in the vocab\n",
        "                records[(items[1],items[2])] = float(items[3])\n",
        "    print(\"Load {} pairs of words for evaluation\".format(len(records)))\n",
        "    return records"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CP9rT0Jfbeuk"
      },
      "source": [
        "### 3.1 Word similarity correlation (2 points)\n",
        "\n",
        "The purpose of this section is to implement the correlation function that compares the predefined scores and the scores computed by cosine similarity. The code of the correlation function is almost done, and the only thing left is the code for computing cosine similarity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "Tazeb8dibeuk"
      },
      "outputs": [],
      "source": [
        "from scipy.stats import pearsonr\n",
        "# TODO: Add necessary packages here\n",
        "from scipy.spatial.distance import cosine\n",
        "def correlation(records, embeddings, vocab):\n",
        "    predefined_scores = []\n",
        "    cossim_scores = []\n",
        "    for (words, sim_score) in records.items():\n",
        "        predefined_scores.append(sim_score)\n",
        "        if words[0] in vocab and words[1] in vocab:\n",
        "            vec1 = embeddings[vocab[words[0]]]\n",
        "            vec2 = embeddings[vocab[words[1]]]\n",
        "            sim = 1 - cosine(vec1, vec2)\n",
        "            #predefined_scores.append(sim_score)\n",
        "            cossim_scores.append(sim)\n",
        "        # ---------------------------------\n",
        "        # TODO: add your code here for computing the cossine similarity\n",
        "        #       between words[0] and words[1], and assign the value to variable \"score\"\n",
        "\n",
        "        # ---------------------------------\n",
        "\n",
        "    corr = pearsonr(predefined_scores, cossim_scores)\n",
        "    return corr"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KzFS-wdnbeuk"
      },
      "source": [
        "Run the following code block to calculate the correlations between pre-defined similarity scores and the cosine similarity scores based on word embeddings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "pzFQqamVbeuk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "05990a11-fbaf-496e-d8e7-4d8b36831579"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Load 151 pairs of words for evaluation\n",
            "The correlation with the LSA embeddings = 0.31924259155765383 with p-value 6.45906927376854e-05\n",
            "Load 151 pairs of words for evaluation\n",
            "The correlation with the skipgram embeddings = 0.34222529117452544 with p-value 1.7003902933571214e-05\n",
            "Skipgram is better than LSA\n"
          ]
        }
      ],
      "source": [
        "records=load_wordpairs(vocab=vocab_lsa)\n",
        "\n",
        "corr_lsa = correlation(records, embeddings_lsa, vocab_lsa)\n",
        "print(\"The correlation with the LSA embeddings = {} with p-value {}\".format(corr_lsa[0], corr_lsa[1]))\n",
        "records=load_wordpairs(vocab=vocab_sg)\n",
        "corr_sg = correlation(records, embeddings_sg, vocab_sg)\n",
        "print(\"The correlation with the skipgram embeddings = {} with p-value {}\".format(corr_sg[0], corr_sg[1]))\n",
        "\n",
        "if corr_lsa[0] > corr_sg[0]:\n",
        "    print(\"LSA is better than Skip-gram\")\n",
        "elif corr_lsa[0] < corr_sg[0]:\n",
        "    print(\"Skipgram is better than LSA\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWOW0SZlbeuk"
      },
      "source": [
        "### 3.2 Analysis of context window size in Skipgram (3 points)\n",
        "\n",
        "With the correlation function, we can analyze the effect of different context window sizes in the Skipgram model. Specifically, please call the previous implementation\n",
        "\n",
        "- `Skipgram(fname, ws, dim=50)` with the context window size `ws` as 3, 6, 9, 12, 15\n",
        "- For each context window size, calculate the correlation using the function `correlation(records, embeddings, vocab)`\n",
        "- **Print out** the fives correlation scores in your final submission: one score per line with the following format\n",
        "<center> ws\\t correlation</center>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "-tv-wIFibeuk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "896e2129-ca52-4396-87c6-bd09e641a4f3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Load 151 pairs of words for evaluation\n",
            "ws= 3 \t The correlation with the skipgram embeddings = 0.34222529117452544 with p-value 1.7003902933571214e-05\n",
            "Load 151 pairs of words for evaluation\n",
            "ws= 6 \t The correlation with the skipgram embeddings = 0.4120620007773998 with p-value 1.4651248219952812e-07\n",
            "Load 151 pairs of words for evaluation\n",
            "ws= 9 \t The correlation with the skipgram embeddings = 0.4398232975955795 with p-value 1.6007465907200303e-08\n",
            "Load 151 pairs of words for evaluation\n",
            "ws= 12 \t The correlation with the skipgram embeddings = 0.4343807165651287 with p-value 2.5101634437980525e-08\n",
            "Load 151 pairs of words for evaluation\n",
            "ws= 15 \t The correlation with the skipgram embeddings = 0.44382442544907136 with p-value 1.1441542431685874e-08\n"
          ]
        }
      ],
      "source": [
        "# TODO: add your code here\n",
        "\n",
        "embeddings_sg, vocab_sg = Skipgram(ws=3)\n",
        "records=load_wordpairs(vocab=vocab_sg)\n",
        "corr_sg = correlation(records, embeddings_sg, vocab_sg)\n",
        "print(\"ws= 3 \\t The correlation with the skipgram embeddings = {} with p-value {}\".format(corr_sg[0], corr_sg[1]))\n",
        "\n",
        "\n",
        "embeddings_sg, vocab_sg = Skipgram(ws=6)\n",
        "records=load_wordpairs(vocab=vocab_sg)\n",
        "corr_sg = correlation(records, embeddings_sg, vocab_sg)\n",
        "print(\"ws= 6 \\t The correlation with the skipgram embeddings = {} with p-value {}\".format(corr_sg[0], corr_sg[1]))\n",
        "\n",
        "\n",
        "\n",
        "embeddings_sg, vocab_sg = Skipgram(ws=9)\n",
        "records=load_wordpairs(vocab=vocab_sg)\n",
        "\n",
        "corr_sg = correlation(records, embeddings_sg, vocab_sg)\n",
        "print(\"ws= 9 \\t The correlation with the skipgram embeddings = {} with p-value {}\".format(corr_sg[0], corr_sg[1]))\n",
        "\n",
        "\n",
        "embeddings_sg, vocab_sg = Skipgram(ws=12)\n",
        "records=load_wordpairs(vocab=vocab_sg)\n",
        "corr_sg = correlation(records, embeddings_sg, vocab_sg)\n",
        "print(\"ws= 12 \\t The correlation with the skipgram embeddings = {} with p-value {}\".format(corr_sg[0], corr_sg[1]))\n",
        "\n",
        "\n",
        "embeddings_sg, vocab_sg = Skipgram(ws=15)\n",
        "records=load_wordpairs(vocab=vocab_sg)\n",
        "corr_sg = correlation(records, embeddings_sg, vocab_sg)\n",
        "print(\"ws= 15 \\t The correlation with the skipgram embeddings = {} with p-value {}\".format(corr_sg[0], corr_sg[1]))\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Sr9rmhRbeuk"
      },
      "source": [
        "Similar experiment can also be conducted on the parameter of negative examples `neg`, but it will not be included in this homework."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "gpuType": "T4"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}