Day2_Word_Embeddings.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jTT7Za7jxHSP"
      },
      "source": [
        "# Day 2. Word Embeddings\n",
        "\n",
        "\n",
        "In this notebook, we will be using the Stanford's pre-trained embedding model **GloVe**. **GloVe** is trained based on the global word-word co-occurrence statistics from a corpus. Researchers were amazed with the linear structures that emerge from the resulting vector space. With this notebook, we hope to showcase you some of the interesting and exciting phenomena related to these word embeddings.  \n",
        "\n",
        "\n",
        "## Agenda\n",
        "\n",
        "- Visualizing word vector representations\n",
        "- Word similarity using word embeddings\n",
        "- Word Analogies\n",
        "\n",
        "![Schematics of word embeddigns and their representation in lower dimensional spaces.](https://miro.medium.com/max/1400/1*sAJdxEsDjsPMioHyzlN3_A.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lL7hn5vJz2qy"
      },
      "source": [
        "## [3 min] Setup \n",
        "\n",
        "In this section, we will install and import the necessary libraries for the execution of this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "Xfgl6e93XfX5"
      },
      "outputs": [],
      "source": [
        "# Dimensionality reduction\n",
        "from sklearn.decomposition import PCA\n",
        "\n",
        "# Word embeddings loading\n",
        "from gensim.models import KeyedVectors\n",
        "import gensim.downloader\n",
        "\n",
        "# Data processing\n",
        "import numpy as np\n",
        "\n",
        "# Data visualization\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Interactive plotting\n",
        "import plotly.graph_objs as go\n",
        "import plotly.io as pio\n",
        "pio.templates.default = \"simple_white\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iCZNcOr7ygo6"
      },
      "source": [
        "We will use the word embeddings that are already available out-of-the-box in the `gensim` library. With the following command, we list all the embeddings that are available for download."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true,
          "base_uri": "https://localhost:8080/"
        },
        "id": "jrS9BuV7xsLg",
        "outputId": "a94d825f-71f3-43b5-ffc7-6000d7a30f58"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "List of available models:\n",
            "\n",
            " -> fasttext-wiki-news-subwords-300\n",
            " -> conceptnet-numberbatch-17-06-300\n",
            " -> word2vec-ruscorpora-300\n",
            " -> word2vec-google-news-300\n",
            " -> glove-wiki-gigaword-50\n",
            " -> glove-wiki-gigaword-100\n",
            " -> glove-wiki-gigaword-200\n",
            " -> glove-wiki-gigaword-300\n",
            " -> glove-twitter-25\n",
            " -> glove-twitter-50\n",
            " -> glove-twitter-100\n",
            " -> glove-twitter-200\n",
            " -> __testing_word2vec-matrix-synopsis\n"
          ]
        }
      ],
      "source": [
        "# Show all available models in gensim\n",
        "print(\"List of available models:\\n\\n ->\", \"\\n -> \".join(list(gensim.downloader.info()[\"models\"].keys())))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "37FYLhCMgNWT"
      },
      "source": [
        "## [5 min] Load Word Embeddings \n",
        "\n",
        "In this notebook, we will be using the Stanford's pre-trained embedding model - **GloVe**. **GloVe** is trained based on the global word-word co-occurrence statistics from a corpus. Researchers were amazed with the linear structures that emerge from the resulting vector space. With this notebook, we hope to showcase you some of the interesting and exciting phenomena related to these word embeddings.  \n",
        "\n",
        "If you'd like to checkout more about the **GloVe** project, check out their [Official Webpage](https://nlp.stanford.edu/projects/glove/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vlwm35Sx023M"
      },
      "source": [
        "**Downloading the *GloVe* representations?** \n",
        "\n",
        "We will use the 50-dimensional **GloVe** vector representation. Since `gensim`, a free open-source Python library for efficiently and effortlessly representing documents as semantic vectors, already provides different **GloVe** models, we will download the 50-dimensional version: `glove-wiki-gigaword-50`.\n",
        "\n",
        "(**Note**: The download may take up to 2 min to conclude.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EsWpNYpjy8hz",
        "outputId": "0db9c583-fce7-4334-cce0-2bac5dee9888"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 14.6 s, sys: 482 ms, total: 15.1 s\n",
            "Wall time: 15.2 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "embeddings = gensim.downloader.load(\"glove-wiki-gigaword-50\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eS9o-UAFALpA",
        "outputId": "b9bb249f-b4bf-4ba2-8204-b689f64e1c9c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of words in vocabulary: 400000\n"
          ]
        }
      ],
      "source": [
        "vocab_words = list(embeddings.vocab.keys())\n",
        "print(\"Number of words in vocabulary:\", len(vocab_words))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bnu4tBfiqnVK"
      },
      "source": [
        "### Helper functions\n",
        "\n",
        "In the next couple of cells, we will define a set of functions that we will use throughout the notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dOPjV2FGAbyk"
      },
      "outputs": [],
      "source": [
        "def train_dim_reducer_for_words(embeddings, words: list, seed: int=12938):\n",
        "  \"\"\"Train a dimensionality reduction algorithm for subset of words.\"\"\"\n",
        "  rand = np.random.default_rng(seed)\n",
        "\n",
        "  # Retrieve the vector representation for each sampled word\n",
        "  matrix = np.array([embeddings[w] for w in words])\n",
        "\n",
        "  # Initialize dimensionality reduction to 3 dimensions (`n_components=3`)\n",
        "  pca = PCA(n_components=3, random_state=rand.integers(0 , 10**6), whiten=True)\n",
        "  # Learn projections to lower dimension\n",
        "  pca.fit(matrix)\n",
        "  print('Variance explained: %.2f' % pca.explained_variance_ratio_.sum())\n",
        "\n",
        "  return pca\n",
        "\n",
        "def train_dim_reducer(embeddings, frac: float=1.0, seed: int=12938):\n",
        "  \"\"\"Train a dimensionality reduction algorithm using the whole \n",
        "  vocabulary representations.\n",
        "  \"\"\"\n",
        "  rand = np.random.default_rng(seed)\n",
        "\n",
        "  vocab = embeddings.vocab.keys()\n",
        "  \n",
        "  if frac < 1.0:\n",
        "    n_samples = int(min(frac * len(vocab), len(vocab)))\n",
        "\n",
        "    # Sample `frac` words from the vocabulary\n",
        "    words = list(rand.choice(vocab, size=n_samples, replace=False))\n",
        "  else:\n",
        "    words = vocab\n",
        "\n",
        "  return train_dim_reducer_for_words(embeddings, words, seed)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RqNBSNLxGIJa"
      },
      "source": [
        "In the next cell, we define two methods for displaying interactive plots of the embeddings:\n",
        "\n",
        "- `plot_word_embeddings2d`: creates 2-dimensional interactive plots of the list of words provided by the user.\n",
        "- `plot_word_embeddings3d`: creates 3-dimensional interactive plots of the words provided by the user."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V9S3iJzEGIRP"
      },
      "outputs": [],
      "source": [
        "def plot_word_embeddings2d(embeddings, dim_reducer, *words_lists: list):\n",
        "  # Flatten list into single list\n",
        "  all_words = []\n",
        "  for lst in words_lists:\n",
        "    all_words.extend(lst)  \n",
        "  \n",
        "  # `emb_matrix` consists of word embeddings stacked vertically.\n",
        "  # the matrix has shape |W| x |E|, where\n",
        "  # - |W| is the number of words in `words`\n",
        "  # - |E| is the dimensionality of the word embeddings\n",
        "  embeddings_matrix = np.array([embeddings[w] for w in all_words])\n",
        "  embeddings_matrix = dim_reducer.transform(embeddings_matrix)\n",
        "    \n",
        "  # ---------------------------------------------\n",
        "  # Create the plot\n",
        "  # ---------------------------------------------\n",
        "  count = 0\n",
        "  traces = []\n",
        "\n",
        "  for words in words_lists:\n",
        "    size = len(words)\n",
        "\n",
        "    trace = go.Scatter(\n",
        "      x = embeddings_matrix[count:count+size,0], \n",
        "      y = embeddings_matrix[count:count+size,1], \n",
        "      text = words,\n",
        "      name = words[0],\n",
        "      textposition = \"top center\",\n",
        "      textfont_size = 20,\n",
        "      mode = 'markers+text',\n",
        "      marker = {\n",
        "          'size': 10,\n",
        "          'opacity': 0.8,\n",
        "          'color': 2\n",
        "    })\n",
        "\n",
        "    count += size\n",
        "    traces.append(trace)\n",
        "\n",
        "  # Configure the layout\n",
        "  layout = go.Layout(\n",
        "      margin = {'l': 0, 'r': 0, 'b': 0, 't': 0},\n",
        "      showlegend=True,\n",
        "      legend=dict(\n",
        "      x=1,\n",
        "      y=0.5,\n",
        "      font=dict(\n",
        "          family=\"Courier New\",\n",
        "          size=25,\n",
        "          color=\"black\"\n",
        "      )),\n",
        "      font = dict(\n",
        "          family = \"Courier New\",\n",
        "          size = 15),\n",
        "      autosize = True,\n",
        "    )\n",
        "\n",
        "  plot_figure = go.Figure(data = traces, layout = layout)\n",
        "  plot_figure.update_yaxes(showgrid=True)\n",
        "  plot_figure.update_xaxes(showgrid=True)\n",
        "  plot_figure.show()\n",
        "\n",
        "\n",
        "def plot_word_embeddings3d(embeddings, dim_reducer, *words_lists: list):\n",
        "  # Flatten list into single list\n",
        "  all_words = []\n",
        "  for lst in words_lists:\n",
        "    all_words.extend(lst)  \n",
        "  \n",
        "  # `emb_matrix` consists of word embeddings stacked vertically.\n",
        "  # the matrix has shape |W| x |E|, where\n",
        "  # - |W| is the number of words in `words`\n",
        "  # - |E| is the dimensionality of the word embeddings\n",
        "  embeddings_matrix = np.array([embeddings.get_vector(w) for w in all_words])\n",
        "  embeddings_matrix = dim_reducer.transform(embeddings_matrix)\n",
        "    \n",
        "  # ---------------------------------------------\n",
        "  # Create the plot\n",
        "  # ---------------------------------------------\n",
        "  count = 0\n",
        "  traces = []\n",
        "\n",
        "  for words in words_lists:\n",
        "    size = len(words)\n",
        "\n",
        "    trace = go.Scatter3d(\n",
        "      x = embeddings_matrix[count:count+size,0], \n",
        "      y = embeddings_matrix[count:count+size,1], \n",
        "      z = embeddings_matrix[count:count+size,2], \n",
        "      text = words,\n",
        "      name = words[0],\n",
        "      textposition = \"top center\",\n",
        "      textfont_size = 10,\n",
        "      mode = 'markers+text',\n",
        "      marker = {\n",
        "          'size': 6,\n",
        "          'opacity': 0.8,\n",
        "          'color': 2\n",
        "    })\n",
        "\n",
        "    count += size\n",
        "    traces.append(trace)\n",
        "\n",
        "  # Configure the layout\n",
        "  layout = go.Layout(\n",
        "      margin = {'l': 0, 'r': 0, 'b': 0, 't': 0},\n",
        "      showlegend=True,\n",
        "      legend=dict(\n",
        "      x=1,\n",
        "      y=0.5,\n",
        "      font=dict(\n",
        "          family=\"Courier New\",\n",
        "          size=20,\n",
        "          color=\"black\"\n",
        "      )),\n",
        "      font = dict(\n",
        "          family = \"Courier New\",\n",
        "          size = 15),\n",
        "      autosize = True,\n",
        "    )\n",
        "\n",
        "  axis_kwargs = dict(\n",
        "    showgrid=True,\n",
        "    gridcolor=\"lightgray\",\n",
        "    showbackground=False,\n",
        "    zerolinecolor=\"black\",\n",
        "  )\n",
        "  plot_figure = go.Figure(data = traces, layout = layout)\n",
        "  plot_figure.update_layout(\n",
        "      scene=dict(xaxis = axis_kwargs, yaxis = axis_kwargs, zaxis = axis_kwargs))\n",
        "  plot_figure.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o43sQw-oSdqL"
      },
      "source": [
        "Now, we will define a method to return the most similar words to a given set of words. This is just a wrapper around the implementation's method to more easily handle scores and the words.\n",
        "\n",
        "Positive words contribute positively towards the similarity, negative words negatively.\n",
        "\n",
        "If you provide a list, this method computes cosine similarity between a simple mean of the projection weight vectors of the given words and the vectors for each word in the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IkpGXmH_Sdvu"
      },
      "outputs": [],
      "source": [
        "def most_similar(embeddings, positive=None, negative=None, with_scores: bool=False, topn=5):\n",
        "  \"\"\"Computes the most similar words to a given set of words. \n",
        "  \n",
        "  Positive words contribute positively towards the similarity, negative words\n",
        "  negatively.\n",
        "\n",
        "  If you provide a list of words, this method computes cosine similarity\n",
        "  between a simple mean of the projection weight vectors of the given words\n",
        "  and the vectors for each word in the model.\n",
        "\n",
        "  Notes\n",
        "  -----\n",
        "  This is just a wrapper around the implementation's method to more easily\n",
        "  handle scores and the words.\n",
        "  \"\"\"\n",
        "  assert positive is not None or negative is not None\n",
        "\n",
        "  words = embeddings.most_similar(positive=positive, negative=negative, topn=topn) \n",
        "  words, scores = zip(*words)\n",
        "  if with_scores:\n",
        "    return words, scores\n",
        "  else:\n",
        "    return words\n",
        "\n",
        "\n",
        "def print_words(words, dec: int=4):\n",
        "  \"\"\"Hacky version: not pretty, do not copy.\"\"\"\n",
        "  if isinstance(words[1], (tuple, list, float)):\n",
        "    for (word, score) in zip(*words):\n",
        "      print(\"-->\", word, \"\\t\", round(score, dec))\n",
        "  else:\n",
        "    for word in words:\n",
        "      print(\"-->\", word)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gKxvoS4zfNgt"
      },
      "source": [
        "The following method computes word analogies according to the following rule:\n",
        "\n",
        "```\n",
        "word4 = word3 + (word2 - word1)\n",
        "```\n",
        "\n",
        "As we will see later on, this will help us retrieve relationships of the form: `'word1' is to 'word2', what 'word3' is to 'word4'`. As an example, you have: \n",
        "`man is to woman, what king is to ______`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JmFzUrMHfN5t"
      },
      "outputs": [],
      "source": [
        "def find_analogy(word1: str, word2: str, word3: str, embeddings=embeddings, with_scores: bool=True, n=2):\n",
        "  \"\"\"Computes the analogy based on the operation word4 = word3 + (word2 - word1).\"\"\"\n",
        "\n",
        "  # For each word, get their word embeddings\n",
        "  emb1 = embeddings[word1]\n",
        "  emb2 = embeddings[word2]\n",
        "  emb3 = embeddings[word3]\n",
        "\n",
        "  result = np.array([-emb1 + emb2 + emb3])\n",
        "  result, scores = most_similar(embeddings, result, with_scores=True, topn=n)\n",
        "\n",
        "  for r, score in zip(result, scores):\n",
        "    message = f\"'{word1}' is to '{word2}', what '{word3}', is to ______?\"\n",
        "    message += f\" ---> {r}\"\n",
        "\n",
        "    if with_scores:\n",
        "      message += f\"\\t{round(score, 4)}\"\n",
        "\n",
        "    print(message)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ONdXRLSK10px"
      },
      "source": [
        "## [5 min] Visualizing Word Embeddings\n",
        "\n",
        "Now that we have the embeddings model, we can map words into a vector space and visualize them.\n",
        "\n",
        "We've implemented two methods that allow you to visualize a lower dimensional representation of the words embeddings. `plot_word_embeddings2d` plots 2D representations, whereas `plot_word_embeddings3d` plots 3D representations. \n",
        "\n",
        "As the inputs, we will need to provide the following:\n",
        "\n",
        "- `embeddings`: the word vector representation that we're using. In this case, the GloVe embeddings object that we've downloaded previously. \n",
        "\n",
        "- `dim_reducer`: a model that is able to transform our word embeddings' high-dimensional representations into low-dimensional representations. We can obtain that either through `train_dim_reducer` or `train_dim_reducer_for_words`.\n",
        "\n",
        "- `list_of_words`: you can pass a single list of words, or multiple lists of words that you wanna plot. The main difference is that per each list you pass as an argument, there should be a different color."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 560
        },
        "id": "fenxdvwg2Qzd",
        "outputId": "3401418b-3b04-4a8d-c81a-e94e885baf26"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Variance explained: 0.13\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"ca70b5ce-0e9c-442d-9db2-3b5262bd0c53\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"ca70b5ce-0e9c-442d-9db2-3b5262bd0c53\")) {                    Plotly.newPlot(                        \"ca70b5ce-0e9c-442d-9db2-3b5262bd0c53\",                        [{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":10},\"mode\":\"markers+text\",\"name\":\"school\",\"text\":[\"school\",\"science\",\"college\",\"university\"],\"textfont\":{\"size\":20},\"textposition\":\"top center\",\"x\":[4.265918731689453,3.872973680496216,4.103238582611084,4.0192084312438965],\"y\":[1.903108835220337,1.9932074546813965,1.9410508871078491,2.652515411376953],\"type\":\"scatter\"},{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":10},\"mode\":\"markers+text\",\"name\":\"fruits\",\"text\":[\"fruits\",\"banana\",\"apple\",\"tomato\"],\"textfont\":{\"size\":20},\"textposition\":\"top center\",\"x\":[2.8073971271514893,2.2448031902313232,3.1770708560943604,2.3701975345611572],\"y\":[0.505411684513092,1.1746950149536133,1.6972848176956177,0.40779995918273926],\"type\":\"scatter\"}],                        {\"autosize\":true,\"font\":{\"family\":\"Courier New\",\"size\":15},\"legend\":{\"font\":{\"color\":\"black\",\"family\":\"Courier New\",\"size\":25},\"x\":1,\"y\":0.5},\"margin\":{\"b\":0,\"l\":0,\"r\":0,\"t\":0},\"showlegend\":true,\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"rgb(36,36,36)\"},\"error_y\":{\"color\":\"rgb(36,36,36)\"},\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"baxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.6}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"rgb(237,237,237)\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"rgb(217,217,217)\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"colorscale\":{\"diverging\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"sequential\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"sequentialminus\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]]},\"colorway\":[\"#1F77B4\",\"#FF7F0E\",\"#2CA02C\",\"#D62728\",\"#9467BD\",\"#8C564B\",\"#E377C2\",\"#7F7F7F\",\"#BCBD22\",\"#17BECF\"],\"font\":{\"color\":\"rgb(36,36,36)\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"white\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"white\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"radialaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"zaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}},\"shapedefaults\":{\"fillcolor\":\"black\",\"line\":{\"width\":0},\"opacity\":0.3},\"ternary\":{\"aaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"baxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"caxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}}},\"yaxis\":{\"showgrid\":true},\"xaxis\":{\"showgrid\":true}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('ca70b5ce-0e9c-442d-9db2-3b5262bd0c53');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Define a set of words to plot:\n",
        "list1 = ['school', 'science', 'college', 'university']\n",
        "list2 = ['fruits', 'banana', 'apple', 'tomato']\n",
        "\n",
        "# Create the dimensionality reductor object\n",
        "dim_reducer = train_dim_reducer(embeddings)\n",
        "\n",
        "# Plot as many list of words as you'd like\n",
        "plot_word_embeddings2d(embeddings, dim_reducer, list1, list2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "a6Gt5hAUEynK",
        "outputId": "660ba1a5-a597-4c78-f9a8-1e84c4951fae"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"b14e0fe8-5eb9-4efc-8736-3270efcf56d6\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"b14e0fe8-5eb9-4efc-8736-3270efcf56d6\")) {                    Plotly.newPlot(                        \"b14e0fe8-5eb9-4efc-8736-3270efcf56d6\",                        [{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":6},\"mode\":\"markers+text\",\"name\":\"school\",\"text\":[\"school\",\"science\",\"college\",\"university\"],\"textfont\":{\"size\":10},\"textposition\":\"top center\",\"x\":[4.265918731689453,3.872973680496216,4.103238582611084,4.0192084312438965],\"y\":[1.903108835220337,1.9932074546813965,1.9410508871078491,2.652515411376953],\"z\":[-1.9257920980453491,-0.8729732632637024,-2.2081496715545654,-3.2734556198120117],\"type\":\"scatter3d\"},{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":6},\"mode\":\"markers+text\",\"name\":\"fruits\",\"text\":[\"fruits\",\"banana\",\"apple\",\"tomato\"],\"textfont\":{\"size\":10},\"textposition\":\"top center\",\"x\":[2.8073971271514893,2.2448031902313232,3.1770708560943604,2.3701975345611572],\"y\":[0.505411684513092,1.1746950149536133,1.6972848176956177,0.40779995918273926],\"z\":[0.12551626563072205,-0.7053747773170471,-0.2518862187862396,-0.6240826845169067],\"type\":\"scatter3d\"}],                        {\"autosize\":true,\"font\":{\"family\":\"Courier New\",\"size\":15},\"legend\":{\"font\":{\"color\":\"black\",\"family\":\"Courier New\",\"size\":20},\"x\":1,\"y\":0.5},\"margin\":{\"b\":0,\"l\":0,\"r\":0,\"t\":0},\"showlegend\":true,\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"rgb(36,36,36)\"},\"error_y\":{\"color\":\"rgb(36,36,36)\"},\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"baxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.6}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"rgb(237,237,237)\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"rgb(217,217,217)\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"colorscale\":{\"diverging\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"sequential\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"sequentialminus\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]]},\"colorway\":[\"#1F77B4\",\"#FF7F0E\",\"#2CA02C\",\"#D62728\",\"#9467BD\",\"#8C564B\",\"#E377C2\",\"#7F7F7F\",\"#BCBD22\",\"#17BECF\"],\"font\":{\"color\":\"rgb(36,36,36)\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"white\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"white\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"radialaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"zaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}},\"shapedefaults\":{\"fillcolor\":\"black\",\"line\":{\"width\":0},\"opacity\":0.3},\"ternary\":{\"aaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"baxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"caxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}}},\"scene\":{\"xaxis\":{\"showgrid\":true,\"gridcolor\":\"lightgray\",\"showbackground\":false,\"zerolinecolor\":\"black\"},\"yaxis\":{\"showgrid\":true,\"gridcolor\":\"lightgray\",\"showbackground\":false,\"zerolinecolor\":\"black\"},\"zaxis\":{\"showgrid\":true,\"gridcolor\":\"lightgray\",\"showbackground\":false,\"zerolinecolor\":\"black\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('b14e0fe8-5eb9-4efc-8736-3270efcf56d6');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_word_embeddings3d(embeddings, dim_reducer, list1, list2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EdqmnPbYMeVa"
      },
      "source": [
        "**Alternatively**: If your representations are not making too much sense, you can opt for reducing the dimensions of only a subset of the words (so that the dimensionality reduction algorithm focuses only on the properties of those word representations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 560
        },
        "id": "u8yPSAMcMd5A",
        "outputId": "e703618c-3fd1-4599-8f2a-0cfb03356ae6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Variance explained: 0.84\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"74615877-e0f6-414d-bfa4-d4682c07ec59\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"74615877-e0f6-414d-bfa4-d4682c07ec59\")) {                    Plotly.newPlot(                        \"74615877-e0f6-414d-bfa4-d4682c07ec59\",                        [{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":10},\"mode\":\"markers+text\",\"name\":\"school\",\"text\":[\"school\",\"science\",\"college\",\"university\"],\"textfont\":{\"size\":20},\"textposition\":\"top center\",\"x\":[0.8400512337684631,0.6656281352043152,1.0103157758712769,1.157885193824768],\"y\":[0.3666805326938629,-0.9653213024139404,0.45567411184310913,-0.24010488390922546],\"type\":\"scatter\"},{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":10},\"mode\":\"markers+text\",\"name\":\"fruits\",\"text\":[\"fruits\",\"banana\",\"apple\",\"tomato\"],\"textfont\":{\"size\":20},\"textposition\":\"top center\",\"x\":[-0.927582323551178,-1.0568091869354248,-0.6381378769874573,-1.0513510704040527],\"y\":[-1.3947771787643433,0.5514733791351318,1.7679357528686523,-0.5415605902671814],\"type\":\"scatter\"}],                        {\"autosize\":true,\"font\":{\"family\":\"Courier New\",\"size\":15},\"legend\":{\"font\":{\"color\":\"black\",\"family\":\"Courier New\",\"size\":25},\"x\":1,\"y\":0.5},\"margin\":{\"b\":0,\"l\":0,\"r\":0,\"t\":0},\"showlegend\":true,\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"rgb(36,36,36)\"},\"error_y\":{\"color\":\"rgb(36,36,36)\"},\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"baxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.6}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"rgb(237,237,237)\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"rgb(217,217,217)\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"colorscale\":{\"diverging\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"sequential\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"sequentialminus\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]]},\"colorway\":[\"#1F77B4\",\"#FF7F0E\",\"#2CA02C\",\"#D62728\",\"#9467BD\",\"#8C564B\",\"#E377C2\",\"#7F7F7F\",\"#BCBD22\",\"#17BECF\"],\"font\":{\"color\":\"rgb(36,36,36)\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"white\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"white\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"radialaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"zaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}},\"shapedefaults\":{\"fillcolor\":\"black\",\"line\":{\"width\":0},\"opacity\":0.3},\"ternary\":{\"aaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"baxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"caxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}}},\"yaxis\":{\"showgrid\":true},\"xaxis\":{\"showgrid\":true}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('74615877-e0f6-414d-bfa4-d4682c07ec59');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Define a set of words to plot:\n",
        "list1 = ['school', 'science', 'college', 'university']\n",
        "list2 = ['fruits', 'banana', 'apple', 'tomato']\n",
        "\n",
        "# Create the dimensionality reductor object\n",
        "dim_reducer2 = train_dim_reducer_for_words(embeddings, list1 + list2)\n",
        "\n",
        "# Plot as many list of words as you'd like\n",
        "plot_word_embeddings2d(embeddings, dim_reducer2, list1, list2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yS4WG7jbMtnB"
      },
      "source": [
        "### Exercise \n",
        "\n",
        "Go ahead an play around with the list of words. You can add as many as you'd like. Note that if you end up selecting words that do not exist you might face errors. **Find some interesting relationships between different words.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UgohCgfZUEH-"
      },
      "outputs": [],
      "source": [
        "list1 = ... # TODO: define your own list of words \n",
        "list2 = ... # TODO: define your own list of words\n",
        "# ... Add as many lists as you want!\n",
        "\n",
        "# You can either re-use the previously created dimensionality reduction object\n",
        "# or create your own, depending on the list of words you specify\n",
        "# dim_reducer = train_dim_reducer_for_words(embeddings, list1 + list2)\n",
        "\n",
        "plot_word_embeddings2d(embeddings, dim_reducer, list1, list2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9e8yBv4M2RE4"
      },
      "source": [
        "## [5 min] Discovering similar words using Word Embeddings\n",
        "\n",
        "\n",
        "We can use the `embeddings` to find the most similar words to either a single word or a list of words! Let's try it out!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rLDy6yEPiyKD",
        "outputId": "13e252d3-8aa3-4826-dfbf-2bb392e57d3a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking for the top 5 most similar words to: summer\n",
            "--> winter \t 0.92\n",
            "--> spring \t 0.8946\n",
            "--> autumn \t 0.8394\n",
            "--> beginning \t 0.8159\n",
            "--> starting \t 0.7925\n",
            "\n",
            "Looking for the top 5 most similar words to: ['summer', 'winter']\n",
            "--> spring \t 0.8966\n",
            "--> autumn \t 0.8601\n",
            "--> beginning \t 0.7695\n",
            "--> day \t 0.7614\n",
            "--> rainy \t 0.7606\n",
            "\n",
            "List of positive words: ['summer', 'winter']\n",
            "List of negative words: ['spring', 'autumn']\n",
            "--> braugher \t 0.5727\n",
            "--> skier \t 0.5636\n",
            "--> snowboarders \t 0.5467\n",
            "--> skiers \t 0.546\n",
            "--> athlete \t 0.5434\n"
          ]
        }
      ],
      "source": [
        "# Number of top similar words to retrieve\n",
        "topn = 5\n",
        "\n",
        "# -----------------------------------------------\n",
        "# Find similar words to a single `word`\n",
        "# -----------------------------------------------\n",
        "word = \"summer\"\n",
        "print(\"Looking for the top\", topn, \"most similar words to:\", word)\n",
        "print_words(most_similar(embeddings, word, topn=topn, with_scores=True))\n",
        "print()\n",
        "\n",
        "# -----------------------------------------------\n",
        "# Find similar words to a list\n",
        "# -----------------------------------------------\n",
        "list_of_words = [\"summer\", \"winter\"]\n",
        "print(\"Looking for the top\", topn, \"most similar words to:\", list_of_words)\n",
        "print_words(most_similar(embeddings, list_of_words, topn=topn, with_scores=True))\n",
        "print()\n",
        "\n",
        "# -----------------------------------------------\n",
        "# Find similar words to a list of words but \n",
        "# not as similar to a seconf list of words.\n",
        "# -----------------------------------------------\n",
        "list_of_pos_words = [\"summer\", \"winter\"]\n",
        "list_of_neg_words = [\"spring\", \"autumn\"]\n",
        "\n",
        "print(\"List of positive words:\", list_of_pos_words)\n",
        "print(\"List of negative words:\", list_of_neg_words)\n",
        "print_words(most_similar(embeddings, list_of_pos_words, list_of_neg_words, topn=topn, with_scores=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ocJTKo_uNlEw"
      },
      "source": [
        "If you do not want to observe the similarity scores? Just remove the flag `with_scores=True` or set it to `False`. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZHzn0ehXNkjL",
        "outputId": "f3c5bc91-83e7-42a4-944b-6d075ee9a03e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking for the top 5 most similar words to: summer\n",
            "--> winter \t 0.92\n",
            "--> spring \t 0.8946\n",
            "--> autumn \t 0.8394\n",
            "--> beginning \t 0.8159\n",
            "--> starting \t 0.7925\n"
          ]
        }
      ],
      "source": [
        "# Number of top similar words to retrieve\n",
        "topn = 5\n",
        "\n",
        "# -----------------------------------------------\n",
        "# Find similar words to a single `word`\n",
        "# -----------------------------------------------\n",
        "word = \"summer\"\n",
        "print(\"Looking for the top\", topn, \"most similar words to:\", word)\n",
        "print_words(most_similar(embeddings, word, topn=topn, with_scores=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HtI4h59TXUX4"
      },
      "source": [
        "### Exercise 1.\n",
        "\n",
        "Play around with the `most_similar` method and find relationships between different pairs of words. Do these make sense to you?\n",
        "\n",
        "\n",
        "You can also use the visualization methods to help you understand better how closely related the words are."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zh0p7NCjXUd1"
      },
      "outputs": [],
      "source": [
        "list_words = ... # TODO:\n",
        "\n",
        "print(most_similar(embeddings, list_words, topn=topn))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w9oeH36lkBon"
      },
      "source": [
        "### Exercise 2. \n",
        "\n",
        "Find interesting mistakes concerning the word embeddings. What was the result you obtained and what were you expecting it to be?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rJq0yThXkBwM"
      },
      "outputs": [],
      "source": [
        "list_words = ... # TODO:\n",
        "\n",
        "print(most_similar(embeddings, list_words, topn=topn))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DOyQdrF8VMOT"
      },
      "source": [
        "In the following sections, we will focus on different types of analogies using word embeddings. We will see how we can leverage the vector space and the vector operations to verify different relationships. \n",
        "\n",
        "## [20 min] Word Analogies\n",
        "\n",
        "We have seen before that word embeddings are nothing but a dense vector representation. When researchers developed these models, they observed interesting linear substructures in this vector space. \n",
        "\n",
        "For example, using vector addition and subtraction, they were able to find relationships between sets of words. The most popularized of which is:\n",
        "\n",
        "> ***man* is to *woman* what *king* is to ____.**\n",
        "\n",
        "Naturally, we expect the answer to be **queen**, but is it? Let's try it out. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_QGC3-IcVMYd",
        "outputId": "cefa0740-1f2e-421a-9a9e-794b20aa599d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "'man' is to 'woman', what 'king', is to ______? ---> king\t0.886\n",
            "'man' is to 'woman', what 'king', is to ______? ---> queen\t0.861\n",
            "'man' is to 'woman', what 'king', is to ______? ---> daughter\t0.7685\n"
          ]
        }
      ],
      "source": [
        "find_analogy(\"man\", \"woman\", \"king\", embeddings=embeddings, n=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mip36gLfh500"
      },
      "source": [
        "The method `find_analogy` performs the following operation: \n",
        "\n",
        "`word4` $\\approx$ (`word2` - `word1`) + `word3`, which in this case becomes `queen` $\\approx$ (`woman` - `man`) + `king`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8nZSRx3IOJbH",
        "outputId": "84ad3d31-dbbc-448a-f3f4-ed0116f79f71"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "'boy' is to 'girl', what 'man', is to ______? ---> woman\t0.9389\n",
            "'boy' is to 'girl', what 'man', is to ______? ---> man\t0.9316\n"
          ]
        }
      ],
      "source": [
        "find_analogy(\"boy\", \"girl\", \"man\", embeddings=embeddings)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JHuL6e5GN12T",
        "outputId": "f0d01b41-3b97-4941-f75e-7a97ffa2e351"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "'paris' is to 'france', what 'berlin', is to ______? ---> germany\t0.9215\n",
            "'paris' is to 'france', what 'berlin', is to ______? ---> denmark\t0.816\n"
          ]
        }
      ],
      "source": [
        "find_analogy(\"paris\", \"france\", \"berlin\", embeddings=embeddings)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tOG2N0YehEix"
      },
      "source": [
        "### Exercise 1.\n",
        "\n",
        "Find other examples of analogies that have not been covered yet.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8rVWb7aAiPci"
      },
      "outputs": [],
      "source": [
        "word1 = ... # TODO\n",
        "word2 = ... # TODO\n",
        "word3 = ... # TODO\n",
        "find_analogy(word1, word2, word3, embeddings=embeddings)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XhuQ0cWNiQFa"
      },
      "source": [
        "### Exercise 2.\n",
        "\n",
        "Can you find any analogy that reflect some biases in the model?\n",
        "\n",
        "\n",
        "(**hint**: what relationship can you find concerning the words `nurse`, `doctor` and `she`)? \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TBnlGeQ0imyB",
        "outputId": "eced0ed0-0ddf-48b6-bffe-e1c14952041b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "'nurse' is to 'doctor', what 'he', is to ______? ---> he\t0.91\n",
            "'nurse' is to 'doctor', what 'he', is to ______? ---> himself\t0.8635\n"
          ]
        }
      ],
      "source": [
        "word1 = ... # TODO\n",
        "word2 = ... # TODO\n",
        "word3 = ... # TODO\n",
        "find_analogy(word1, word2, word3, embeddings=embeddings)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "biBptok4imsC"
      },
      "source": [
        "### Exercise 3. \n",
        "\n",
        "Find examples of analogies using verb tenses. Do you expect these to be captured by the model? "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xbRBprTB0jjn"
      },
      "outputs": [],
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYGXbSS8i04H"
      },
      "source": [
        "### Exercise 4. \n",
        "\n",
        "What happens if you visualize some of these words in a lower dimensional space? Do you expect some of them to be near each other or farther apart?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oLRn8B36zNKO"
      },
      "outputs": [],
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MNnIO_DCzClk"
      },
      "source": [
        "### Trying your own analogies\n",
        "\n",
        "If you want to try out your own vector operations, try following this recipe:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k9QleSvfzA8l"
      },
      "outputs": [],
      "source": [
        "# To do any analogies you desire, you can proceed as follows:\n",
        "# Step 1. Get the word vector reprensentation\n",
        "word1 = \"king\"\n",
        "rep1 = embeddings[word1]\n",
        "\n",
        "word2 = \"man\"\n",
        "rep2 = embeddings[word2]\n",
        "\n",
        "# Step2. Specify the operation between the words \n",
        "rep = rep1 + rep2\n",
        "result = np.array([rep])\n",
        "\n",
        "print_words(most_similar(embeddings, result, with_scores=True, topn=5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "83VlVPwCzDhD"
      },
      "source": [
        "## [Optionally] Working with embeddings on other languages... \n",
        "\n",
        "All our experimentes so far considered *English* as the main language. However, one question you might have is whether these relationships also verify for other languages. \n",
        "\n",
        "In this section, we would like you to exploit the embeddings available in other languages. To do so, we present you with two options:\n",
        "\n",
        "1. Pick **monolingual** word vector representation: these were trained on other languages from scratch and might lead to completely different relationships.\n",
        "\n",
        "2. Pick **cross-languages** word vector representation: these are given words in two different languages and trained to represent the two languages in the same vector space.\n",
        "\n",
        "\n",
        "If you'd like to pick a **monolingual** you can keep reading the next section: **Working with monolingual embeddings**. However, if you prefer the **cross-lingual** version, you should skip to the next session **Working with cross-lingual embeddings**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ASD5wWIoeE6Q"
      },
      "source": [
        "### Working with Monolingual Embeddings\n",
        "\n",
        "If you know multiple languages, you might be wondering what relationships word embeddings in your language exhibit. \n",
        "\n",
        "1. Browse [this list of pretrained vectors](https://fasttext.cc/docs/en/pretrained-vectors.html) and identify the language you want to explore.\n",
        "2. Copy the **text** link and assign it to the variable **url**. We have an example in the code cell below.\n",
        "3. Load the vector representation into a Python variable (we'll use `gensim.KeyedVectors.load_word2vec_format` for that."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XLqbieIdwZbl"
      },
      "outputs": [],
      "source": [
        "url = ... # TODO: Paste the URL here\n",
        "\n",
        "## Example: If we were to consider portuguese (pt) embeddings, \n",
        "## the url would be:\n",
        "## url = \"https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.pt.vec\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7kef7jmBssaB",
        "outputId": "4487d28e-5ec2-4b31-d47b-fa16f21aaff6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100 1488M  100 1488M    0     0  23.5M      0  0:01:03  0:01:03 --:--:-- 26.4M\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "# downloads the specified url and creates a file named \"wiki.vec\"\n",
        "!curl -Lo wiki.vec {url}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wXnP9ju8zCrQ",
        "outputId": "4a774f53-5055-4d98-fa6c-4b7603ca59b2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Size of the vocabulary: 592108\n"
          ]
        }
      ],
      "source": [
        "# Load the embeddings\n",
        "embeddings_lang = KeyedVectors.load_word2vec_format(\"wiki.vec\")\n",
        "print(\"Size of the vocabulary:\", len(embeddings_lang.vocab))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A0keUNdvqjVV"
      },
      "source": [
        "\n",
        "#### Exercise\n",
        "\n",
        "Do the relationships you found previously verify for a different language (e.g., spanish or chinese)? Can you find new patterns on a different language? "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_vLqwuREvvn3"
      },
      "outputs": [],
      "source": [
        "word = ... # TODO: define a word in the language you chose\n",
        "most_similar(embeddings_lang, word)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OmrCKxZLVW-k"
      },
      "source": [
        "### 2. Working with cross-lingual embeddings \n",
        "\n",
        "\n",
        "1. Browse to this [list of pretrained vectors](https://github.com/facebookresearch/MUSE#multilingual-word-embeddings) and identify the language you want to explore. These multilingual word embeddings will contain most words of the language you pick, as well as some words of other languages.\n",
        "2. Copy the **text** link and assign it to the variable **url**. We have an example in the code cell below, where we pick the Portuguese multilingual embeddings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YBlGEh-O3iKz",
        "outputId": "f76252ef-cbf1-4a3d-b857-fbe3cee02c56"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 2.42 ms, sys: 42 µs, total: 2.47 ms\n",
            "Wall time: 2.48 ms\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "%%capture\n",
        "# TODO: Change this URL with the one you copy from\n",
        "# https://github.com/facebookresearch/MUSE#multilingual-word-embeddings\n",
        "url = \"https://dl.fbaipublicfiles.com/arrival/vectors/wiki.multi.pt.vec\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aJHyyPGo3iR0",
        "outputId": "e1d0b90a-8d0d-423c-fc94-65089b6cbc71"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 35.2 s, sys: 1.46 s, total: 36.6 s\n",
            "Wall time: 54.1 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "%%capture\n",
        "# Retrieve the specified multilingual vector space\n",
        "!curl -Lo wiki.multi.vec {url}\n",
        "\n",
        "# Load the embeddings\n",
        "embeddings_multi_lang = KeyedVectors.load_word2vec_format(\"wiki.multi.vec\")\n",
        "print(\"Size of the vocabulary:\", len(embeddings_multi_lang.vocab))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i5-kkzkhnkS5"
      },
      "source": [
        "Use the methods explored earlier in the class to study the relationships in your selected language. We added some examples in Portuguese... Feel free to modify them as you please!\n",
        "\n",
        "\n",
        "You'll probably notice that the english representations in these cross-lingual representations are not as strong as the multilingual."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nt-7Lwic4_LS",
        "outputId": "38c0f333-fbbf-4bfa-ba16-9b6bed5b0a04"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--> s/mileage \t 0.5829\n",
            "--> musume \t 0.56\n",
            "--> berryz \t 0.5411\n",
            "--> goodbye \t 0.531\n",
            "--> project \t 0.5243\n",
            "\n",
            "--> belanidia \t 0.6121\n",
            "--> gostaria \t 0.5944\n",
            "--> desculpe \t 0.594\n",
            "--> criei \t 0.5937\n",
            "--> prezados \t 0.5904\n"
          ]
        }
      ],
      "source": [
        "print_words(most_similar(embeddings_multi_lang, \"hello\", with_scores=True))\n",
        "print()\n",
        "print_words(most_similar(embeddings_multi_lang, \"olá\", with_scores=True))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JoZA_p5X-2n3",
        "outputId": "fd081ac9-f5ad-46f0-c25b-4bc4d1fec3d3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--> friends \t 0.7429\n",
            "--> boyfriend \t 0.7327\n",
            "--> friendly \t 0.656\n",
            "--> girlfriend \t 0.6395\n",
            "--> everyone \t 0.6392\n",
            "\n",
            "--> namorador \t 0.6774\n",
            "--> amigos \t 0.6656\n",
            "--> companheiro \t 0.6611\n",
            "--> colega \t 0.6526\n",
            "--> admirador \t 0.6438\n"
          ]
        }
      ],
      "source": [
        "print_words(most_similar(embeddings_multi_lang, \"friend\", with_scores=True))\n",
        "print()\n",
        "print_words(most_similar(embeddings_multi_lang, \"amigo\", with_scores=True))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_2OzoNsF5YOS",
        "outputId": "dc242d78-f0eb-4c8a-a18c-ac6373423cc9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--> mwlanguage \t 0.7947\n",
            "--> languages \t 0.7731\n",
            "--> línguagem \t 0.7462\n",
            "--> interlíngua \t 0.7234\n",
            "--> falada \t 0.7188\n"
          ]
        }
      ],
      "source": [
        "print_words(most_similar(embeddings_multi_lang, [\"language\", u\"língua\"], with_scores=True))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o1jMFzPMAA63",
        "outputId": "92e15595-194d-41c8-e494-e5d512253f56"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "'homem' is to 'rei', what 'mulher', is to ______? ---> rei\t0.7485\n",
            "'homem' is to 'rei', what 'mulher', is to ______? ---> rainha\t0.6603\n"
          ]
        }
      ],
      "source": [
        "find_analogy(\"homem\", \"rei\", \"mulher\", embeddings_multi_lang)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gI78-_2GAH9p",
        "outputId": "6b42c729-8744-476a-a09f-aac4f6ba5fc2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "'homem' is to 'enfermeiro', what 'mulher', is to ______? ---> enfermeiro\t0.774\n",
            "'homem' is to 'enfermeiro', what 'mulher', is to ______? ---> enfermeira\t0.7595\n"
          ]
        }
      ],
      "source": [
        "find_analogy(\"homem\", \"enfermeiro\", \"mulher\", embeddings_multi_lang)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ShaygwqwANqg",
        "outputId": "b29a0827-a4df-45d4-d9f7-1d3ee5032267"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "comer is to comi what beber is to ______?\n",
            "--> bebi\n"
          ]
        }
      ],
      "source": [
        "find_analogy(\"comer\", \"comi\", \"beber\", embeddings_multi_lang)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "ZSdOAjKp5mxP",
        "outputId": "1c2a9107-1934-4700-c78f-31a545ddd77e"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"d8ff1388-06b5-4005-9365-ff6f1f2163b4\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"d8ff1388-06b5-4005-9365-ff6f1f2163b4\")) {                    Plotly.newPlot(                        \"d8ff1388-06b5-4005-9365-ff6f1f2163b4\",                        [{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":10},\"mode\":\"markers+text\",\"name\":\"man\",\"text\":[\"man\",\"king\",\"woman\",\"queen\",\"eat\",\"ate\"],\"textfont\":{\"size\":20},\"textposition\":\"top center\",\"x\":[-0.09011346846818924,1.4347633123397827,0.2827252745628357,1.4872355461120605,0.024416033178567886,-0.7378502488136292],\"y\":[-1.5305429697036743,-0.11421270668506622,-1.3770084381103516,-0.5447651147842407,-0.8267617225646973,0.6408439874649048],\"type\":\"scatter\"},{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":10},\"mode\":\"markers+text\",\"name\":\"homem\",\"text\":[\"homem\",\"rei\",\"mulher\",\"rainha\",\"comer\",\"comi\"],\"textfont\":{\"size\":20},\"textposition\":\"top center\",\"x\":[-1.000396728515625,0.722070038318634,-0.5027159452438354,0.9757102131843567,-1.2839289903640747,-1.3119151592254639],\"y\":[-0.27335259318351746,1.6902861595153809,0.026895076036453247,1.4217140674591064,0.5147756934165955,0.3721289336681366],\"type\":\"scatter\"}],                        {\"autosize\":true,\"font\":{\"family\":\"Courier New\",\"size\":15},\"legend\":{\"font\":{\"color\":\"black\",\"family\":\"Courier New\",\"size\":25},\"x\":1,\"y\":0.5},\"margin\":{\"b\":0,\"l\":0,\"r\":0,\"t\":0},\"showlegend\":true,\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"rgb(36,36,36)\"},\"error_y\":{\"color\":\"rgb(36,36,36)\"},\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"baxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.6}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"rgb(237,237,237)\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"rgb(217,217,217)\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"colorscale\":{\"diverging\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"sequential\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"sequentialminus\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]]},\"colorway\":[\"#1F77B4\",\"#FF7F0E\",\"#2CA02C\",\"#D62728\",\"#9467BD\",\"#8C564B\",\"#E377C2\",\"#7F7F7F\",\"#BCBD22\",\"#17BECF\"],\"font\":{\"color\":\"rgb(36,36,36)\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"white\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"white\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"radialaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"zaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}},\"shapedefaults\":{\"fillcolor\":\"black\",\"line\":{\"width\":0},\"opacity\":0.3},\"ternary\":{\"aaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"baxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"caxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}}},\"yaxis\":{\"showgrid\":true},\"xaxis\":{\"showgrid\":true}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('d8ff1388-06b5-4005-9365-ff6f1f2163b4');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "src_words = ['man', 'king', 'woman', 'queen', 'eat', 'ate']\n",
        "tgt_words = ['homem', 'rei', 'mulher', 'rainha',  'comer', 'comi']\n",
        "\n",
        "dim_reducer = train_dim_reducer_for_words(embeddings_multi_lang, src_words + tgt_words)\n",
        "plot_word_embeddings2d(embeddings_multi_lang, dim_reducer, src_words, tgt_words)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "tdMOV_EtXLR4",
        "outputId": "ca8192a6-f12e-4a30-f337-7bd73c2ea7e3"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"ac99b694-54ec-4609-97fa-3d99bbc48f09\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"ac99b694-54ec-4609-97fa-3d99bbc48f09\")) {                    Plotly.newPlot(                        \"ac99b694-54ec-4609-97fa-3d99bbc48f09\",                        [{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":6},\"mode\":\"markers+text\",\"name\":\"car\",\"text\":[\"car\",\"cat\",\"deep\",\"feline\"],\"textfont\":{\"size\":10},\"textposition\":\"top center\",\"x\":[0.6276847124099731,-0.21949312090873718,1.4379241466522217,-0.5606994032859802],\"y\":[1.7497586011886597,0.2958885431289673,-0.8024708032608032,-0.13515689969062805],\"z\":[0.10755794495344162,1.83993399143219,1.1426821947097778,0.00809023529291153],\"type\":\"scatter3d\"},{\"marker\":{\"color\":2,\"opacity\":0.8,\"size\":6},\"mode\":\"markers+text\",\"name\":\"carro\",\"text\":[\"carro\",\"gato\",\"gata\",\"profundo\",\"felino\"],\"textfont\":{\"size\":10},\"textposition\":\"top center\",\"x\":[0.561765730381012,-1.1935054063796997,-1.040311574935913,1.2607461214065552,-0.8741098046302795],\"y\":[1.348276138305664,-0.23859761655330658,-0.3060559332370758,-1.3990671634674072,-0.5125753879547119],\"z\":[-1.2408404350280762,-0.35035765171051025,0.2211955040693283,-1.0791748762130737,-0.6490862369537354],\"type\":\"scatter3d\"}],                        {\"autosize\":true,\"font\":{\"family\":\"Courier New\",\"size\":15},\"legend\":{\"font\":{\"color\":\"black\",\"family\":\"Courier New\",\"size\":20},\"x\":1,\"y\":0.5},\"margin\":{\"b\":0,\"l\":0,\"r\":0,\"t\":0},\"showlegend\":true,\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"rgb(36,36,36)\"},\"error_y\":{\"color\":\"rgb(36,36,36)\"},\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"baxis\":{\"endlinecolor\":\"rgb(36,36,36)\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"rgb(36,36,36)\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"line\":{\"color\":\"white\",\"width\":0.6}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"},\"colorscale\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"rgb(237,237,237)\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"rgb(217,217,217)\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":1,\"tickcolor\":\"rgb(36,36,36)\",\"ticks\":\"outside\"}},\"colorscale\":{\"diverging\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"sequential\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]],\"sequentialminus\":[[0.0,\"#440154\"],[0.1111111111111111,\"#482878\"],[0.2222222222222222,\"#3e4989\"],[0.3333333333333333,\"#31688e\"],[0.4444444444444444,\"#26828e\"],[0.5555555555555556,\"#1f9e89\"],[0.6666666666666666,\"#35b779\"],[0.7777777777777778,\"#6ece58\"],[0.8888888888888888,\"#b5de2b\"],[1.0,\"#fde725\"]]},\"colorway\":[\"#1F77B4\",\"#FF7F0E\",\"#2CA02C\",\"#D62728\",\"#9467BD\",\"#8C564B\",\"#E377C2\",\"#7F7F7F\",\"#BCBD22\",\"#17BECF\"],\"font\":{\"color\":\"rgb(36,36,36)\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"white\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"white\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"radialaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"zaxis\":{\"backgroundcolor\":\"white\",\"gridcolor\":\"rgb(232,232,232)\",\"gridwidth\":2,\"linecolor\":\"rgb(36,36,36)\",\"showbackground\":true,\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}},\"shapedefaults\":{\"fillcolor\":\"black\",\"line\":{\"width\":0},\"opacity\":0.3},\"ternary\":{\"aaxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"baxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"},\"bgcolor\":\"white\",\"caxis\":{\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"rgb(232,232,232)\",\"linecolor\":\"rgb(36,36,36)\",\"showgrid\":false,\"showline\":true,\"ticks\":\"outside\",\"title\":{\"standoff\":15},\"zeroline\":false,\"zerolinecolor\":\"rgb(36,36,36)\"}}},\"scene\":{\"xaxis\":{\"showgrid\":true,\"gridcolor\":\"lightgray\",\"showbackground\":false,\"zerolinecolor\":\"black\"},\"yaxis\":{\"showgrid\":true,\"gridcolor\":\"lightgray\",\"showbackground\":false,\"zerolinecolor\":\"black\"},\"zaxis\":{\"showgrid\":true,\"gridcolor\":\"lightgray\",\"showbackground\":false,\"zerolinecolor\":\"black\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('ac99b694-54ec-4609-97fa-3d99bbc48f09');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "src_words = ['car', 'cat', 'deep', 'feline']\n",
        "tgt_words = ['carro', 'gato', 'gata', 'profundo', 'felino']\n",
        "\n",
        "dim_reducer = train_dim_reducer_for_words(embeddings_multi_lang, src_words + tgt_words)\n",
        "plot_word_embeddings3d(embeddings_multi_lang, dim_reducer, src_words, tgt_words)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qb1psoaHzq0J"
      },
      "source": [
        "# Resources\n",
        "\n",
        "This notebook is inspired on:\n",
        "\n",
        "- [How to add large datasets to google drive](https://www.aboutdatablog.com/post/how-to-successfully-add-large-data-sets-to-google-drive-and-use-them-in-google-colab) by Magdalena Kokiewicz.\n",
        "- [Visualizing Word Embedding with PCA and t-SNE](https://towardsdatascience.com/visualizing-word-embedding-with-pca-and-t-sne-961a692509f5) by Ruben Winastwan.\n",
        "- [Looking up word analogies using GloVe along the Fun-Boring axis](https://towardsdatascience.com/glove-fun-boring-664fe0717c4c) by Ludi Rehak\n",
        "- [Solving Analogies by representing Words as Vectors](https://medium.datadriveninvestor.com/solving-analogies-using-word2vec-13b9e01eca1) by Rahul Rathi\n",
        "- [Wikipedia2Vec](https://wikipedia2vec.github.io/wikipedia2vec/pretrained/)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Day2-Word-Embeddings.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}