Beam Search and Sentiment classifier.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NL2DS-F2021-Assignment-6_Suheng Qian.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2FuO82RTBftK"
      },
      "source": [
        "# 1. Language Modeling\n",
        "\n",
        "In this part, let's generate text using a trigram language model.\n",
        "\n",
        "Go to https://drive.google.com/drive/folders/1pR0koayRSgXfTD72HZUHN14uec0SrnXy?usp=sharing and click add shortcut to drive. This will add the data required for this problem set to your Google drive.\n",
        "\n",
        "<img src=\"https://drive.google.com/uc?id=1LqHisiziX8Ri94Xs6Cv8mhx6vivFM3kS\" alt=\"Drawing\" height=\"300\"/>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UtZEcHthBeXz"
      },
      "source": [
        "Run the below code snippet. It will generate a URL which generates an authorization code.* Enter it below to give Colab access to your Google drive. \n",
        "\n",
        "*Copy function may not work. If so, manually copy the authorization code."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KW-dce7oJlyr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5b606fe8-f4d4-49b6-f4a4-c8c40b2c294a"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive/', force_remount=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive/\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ni2pYuuQKaHY"
      },
      "source": [
        "When you run the `ls` command below, you should see these folders.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zYENtyc7SOxA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "86de0ac7-c280-410c-8b97-48571d13f722"
      },
      "source": [
        "!ls \"/content/drive/My Drive/nl2ds\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "semantic-parser  tweets\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K2Y7I_9lPoZS"
      },
      "source": [
        "Let's load the trigrams first. You can change the below code as you see fit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gZMOmElPSPHk"
      },
      "source": [
        "from math import log\n",
        "\n",
        "bigram_prefix_to_trigram = {}\n",
        "bigram_prefix_to_trigram_weights = {}\n",
        "\n",
        "lines = open(\"/content/drive/My Drive/nl2ds/tweets/covid-tweets-2020-08-10-2020-08-21.trigrams.txt\").readlines()\n",
        "for line in lines:\n",
        "  word1, word2, word3, count = line.strip().split()\n",
        "  if (word1, word2) not in bigram_prefix_to_trigram:\n",
        "    bigram_prefix_to_trigram[(word1, word2)] = []\n",
        "    bigram_prefix_to_trigram_weights[(word1, word2)] = []\n",
        "  bigram_prefix_to_trigram[(word1, word2)].append(word3)\n",
        "  bigram_prefix_to_trigram_weights[(word1, word2)].append(int(count))\n",
        "\n",
        "# freeup memory\n",
        "lines = None"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X48i3rarPzd8"
      },
      "source": [
        "## Problem 1.1: Retrieve top next words and their probability given a bigram prefix.\n",
        "\n",
        "For the following prefixes **word1=middle, word2=of, and n=10**, the output is:\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "a 0.807981220657277\n",
        "the 0.06948356807511737\n",
        "pandemic 0.023943661971830985\n",
        "this 0.016901408450704224\n",
        "an 0.0107981220657277\n",
        "...\n",
        "...\n",
        "...\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XYhal88xSYow",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cb6c2ddb-0445-45ef-c255-6c6c36b9f3f1"
      },
      "source": [
        "def top_next_word(word1, word2, n=10):\n",
        "  # write your code here\n",
        "  top_words=bigram_prefix_to_trigram[(word1, word2)][0:n]\n",
        "  top_weights=bigram_prefix_to_trigram_weights[(word1, word2)][0:n]\n",
        "  top_probs=[]\n",
        "  for weight in top_weights:\n",
        "    probability=weight/sum(bigram_prefix_to_trigram_weights[(word1, word2)])\n",
        "    top_probs.append(probability)\n",
        "  prob_dict=dict(zip(top_words, top_probs))\n",
        "  words=[k  for  k in  prob_dict.keys()]\n",
        "  probs=[v  for  v in  prob_dict.values()]\n",
        "  return words,probs\n",
        "next_words, probs = top_next_word(\"middle\", \"of\", 10)\n",
        "for word, prob in zip(next_words,probs):\n",
        "  print(word, prob)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "a 0.807981220657277\n",
            "the 0.06948356807511737\n",
            "pandemic 0.023943661971830985\n",
            "this 0.016901408450704224\n",
            "an 0.0107981220657277\n",
            "covid 0.009389671361502348\n",
            "nowhere 0.008450704225352112\n",
            "it 0.004694835680751174\n",
            "lockdown 0.002347417840375587\n",
            "summer 0.002347417840375587\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gok10i2dSHXB"
      },
      "source": [
        "## Problem 1.2: Sampling n words\n",
        "\n",
        "Sample next n words given a bigram prefix. Use the probablity distribution defined by the frequency counts. Functions like **numpy.random.choice** will be useful here. Sample without repitition, otherwise all your samples will contain the most frequent trigram.\n",
        "\n",
        "\n",
        "For the following prefixes **word1=middle, word2=of, and n=10**, the output could be as follows (our outputs may differ): \n",
        "\n",
        "```\n",
        "a 0.807981220657277\n",
        "pandemic 0.023943661971830985\n",
        "nowhere 0.008450704225352112\n",
        "the 0.06948356807511737\n",
        "...\n",
        "...\n",
        "...\n",
        "...\n",
        "...\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7OzYJoYfUaom",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "732fd519-0a96-4431-b568-8d15beecbcf9"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "def sample_next_word(word1, word2, n=10):\n",
        "  # write your code here\n",
        "  top_words=bigram_prefix_to_trigram[(word1, word2)]\n",
        "  top_weights=bigram_prefix_to_trigram_weights[(word1, word2)]\n",
        "\n",
        "  top_probs=[]\n",
        "  prob_dict={}\n",
        "  for weight in top_weights:\n",
        "      probability=weight/sum(bigram_prefix_to_trigram_weights[(word1, word2)])\n",
        "      top_probs.append(probability)\n",
        "  prob_dict=dict(zip(top_words, top_probs))\n",
        "  \n",
        "  try:\n",
        "        next_words = np.random.choice(top_words, n, p=top_probs,replace=False)\n",
        "  except ValueError:\n",
        "        next_words = np.random.choice(top_words, len(top_words), p=top_probs,replace=False)\n",
        "  \n",
        "  probs=[prob_dict[word] for word in next_words]\n",
        "  return next_words,probs\n",
        "next_words, probs = sample_next_word(\"<BOS2>\", \"trump\", 10)\n",
        "for word, prob in zip(next_words, probs):\n",
        "  print(word, prob)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "criticizes 0.0013550135501355014\n",
            "proposes 0.0013550135501355014\n",
            "'s 0.06639566395663957\n",
            "could 0.0033875338753387536\n",
            "covid 0.0020325203252032522\n",
            "news 0.0033875338753387536\n",
            "is 0.0989159891598916\n",
            "keeps 0.0033875338753387536\n",
            "should 0.008807588075880758\n",
            "continues 0.0040650406504065045\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JyYenU8H-fIR"
      },
      "source": [
        "## Problem 1.3: Generate sentences starting with a prefix\n",
        "\n",
        "Generates n-sentences starting with a given sentence prefix. Use [beam search](https://en.wikipedia.org/wiki/Beam_search) to generate multiple sentences. Depending on which method you use to generate next word, you will get different outputs. When you generate <EOS> in a path, stop exploring that path. If you are not careful with your implementation, you may end up in an infinite loop.\n",
        "\n",
        "If you use the method `word_generator=top_next_word`, `beam=10` and prefix is `<BOS1> <BOS2> trump`, your output is as follows:\n",
        "```\n",
        "<BOS1> <BOS2> trump eyes new unproven coronavirus treatment URL <EOS> 0.00021893147502903603\n",
        "<BOS1> <BOS2> trump eyes new unproven coronavirus cure URL <EOS> 0.0001719607222046247\n",
        "<BOS1> <BOS2> trump eyes new unproven virus cure promoted by mypillow ceo over unproven therapeutic URL <EOS> 9.773272077557522e-05\n",
        "...\n",
        "...\n",
        "...\n",
        "```\n",
        "\n",
        "\n",
        "If you use the method `word_generator=top_next_word`, `beam=10` and prefix is `<BOS1> <BOS2> biden`, your output is as follows:\n",
        "```\n",
        "<BOS1> <BOS2> biden calls for a 30 bonus URL #cashgem #cashappfriday #stayathome <EOS> 0.0002495268686322749\n",
        "<BOS1> <BOS2> biden says all u.s. governors should mandate masks <EOS> 1.6894510541025754e-05\n",
        "<BOS1> <BOS2> biden says all u.s. governors question cost of a pandemic <EOS> 8.777606198953028e-07\n",
        "...\n",
        "...\n",
        "...\n",
        "```\n",
        "\n",
        "\n",
        "If you use the method `word_generator=sample_next_word`, `beam=10` and prefix is `<BOS1> <BOS2> trump`, your output may look as follows (since this is sampling, our outputs will difer):\n",
        "\n",
        "```\n",
        "<BOS1> <BOS2> trump signs executive orders URL <EOS> 7.150992253427233e-05\n",
        "<BOS1> <BOS2> trump signs executive actions URL <EOS> 7.117242889600614e-05\n",
        "<BOS1> <BOS2> trump news president attacked over it <EOS> 1.0546494007903964e-05\n",
        "<BOS1> <BOS2> trump news president attacked over executive orders URL <EOS> 1.0126405114118984e-05\n",
        "```\n",
        "\n",
        "If you use the method `word_generator=sample_next_word`, `beam=10` and prefix is `<BOS1> <BOS2> biden`, your output may look as follows:\n",
        "\n",
        "```\n",
        "<BOS1> <BOS2> biden harris 2020 <EOS> 0.0015758924114719264\n",
        "<BOS1> <BOS2> biden harris 2020 URL <EOS> 0.0006443960952032196\n",
        "<BOS1> <BOS2> biden calls for evictions ban so marylander 's do it URL <EOS> 4.105215709355001e-07\n",
        "<BOS1> <BOS2> biden calls for evictions ban so marylander 's do our best to stay home <EOS> 1.3158806336098573e-09\n",
        "...\n",
        "...\n",
        "...\n",
        "...\n",
        "...\n",
        "```\n",
        "\n",
        "Hope you see that sampling gives different outputs compared to deterministically picking the top n-words.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "50aa1mbu3sxC"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "40kW0joweXFO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "130fd7cb-5557-4eb6-eab5-c91065ea866e"
      },
      "source": [
        "import heapq\n",
        " \n",
        "class Beam(object):\n",
        "#For comparison of prefixes, the tuple (prefix_probability, complete_sentence) is used.\n",
        "#This is so that if two prefixes have equal probabilities then a complete sentence is preferred over an incomplete one since (0.5, False) < (0.5, True)\n",
        " \n",
        "    def __init__(self, beam_width):\n",
        "        self.heap = list()\n",
        "        self.beam_width = beam_width\n",
        " \n",
        "    def add(self, prob, complete, prefix):\n",
        "        heapq.heappush(self.heap, (prob, complete, prefix))\n",
        "        if len(self.heap) > self.beam_width:\n",
        "            heapq.heappop(self.heap)\n",
        "     \n",
        "    def __iter__(self):\n",
        "        return iter(self.heap)\n",
        "def generate_sentences(prefix, sampler, beam=10,clip_len=-1):\n",
        "  # write your code\n",
        "  prefix_split = prefix.split(\" \")\n",
        "  prev_beam = Beam(beam)\n",
        "  sentence=[\"<BOS2>\", prefix_split[len(prefix_split) - 1]]\n",
        "  prev_beam.add(1.0, False, sentence)\n",
        "  while True:\n",
        "      curr_beam = Beam(beam)\n",
        "      return_sents=[]\n",
        "      return_probs=[]\n",
        "      for (prefix_prob, complete, sentence) in prev_beam:\n",
        "          if complete == True:\n",
        "              curr_beam.add(prefix_prob, complete, sentence)\n",
        "              continue\n",
        "          else:\n",
        "            last_word = sentence[-1]\n",
        "            second_last = sentence[-2]\n",
        "            next_words,next_probs=sampler(second_last,last_word,10)\n",
        "            for i in range(len(next_words)):\n",
        "                copy_sentence=sentence.copy()\n",
        "                copy_sentence.append(next_words[i])\n",
        "                complete = False\n",
        "                if next_words[i] == \"<EOS>\": #If next word is EOS, change the complete status\n",
        "                    complete=True\n",
        "                    curr_beam.add(prefix_prob*next_probs[i], complete, copy_sentence)\n",
        "                else: #if next word is not EOS, continue \n",
        "                    curr_beam.add(prefix_prob*next_probs[i], complete, copy_sentence) \n",
        "      top_beam = heapq.nlargest(10, curr_beam)    \n",
        "      begin=['<BOS1>']#Add the first BOS since it wasn't included at first\n",
        "      for best_prob, best_complete, best_sentence in top_beam:\n",
        "        if best_complete == True:\n",
        "          if best_sentence not in return_sents:\n",
        "            return_sents.append(' '.join(begin+best_sentence))\n",
        "            return_probs.append(best_prob)\n",
        "      if len(return_sents) >= 10:\n",
        "            return return_sents,return_probs\n",
        "      prev_beam = curr_beam\n",
        "\n",
        "     \n",
        "\n",
        "sentences, probs = generate_sentences(prefix=\"<BOS1> <BOS2> trump\", beam=10, sampler=top_next_word)\n",
        "for sent, prob in zip(sentences, probs):\n",
        "  print(sent, prob)\n",
        "print(\"#########################\\n\")\n",
        "\n",
        "sentences, probs = generate_sentences(prefix=\"<BOS1> <BOS2> biden\", beam=10, sampler=top_next_word)\n",
        "for sent, prob in zip(sentences, probs):\n",
        "  print(sent, prob)\n",
        "print(\"#########################\\n\")\n",
        "\n",
        "sentences, probs = generate_sentences(prefix=\"<BOS1> <BOS2> trump\", beam=10, sampler=sample_next_word)\n",
        "for sent, prob in zip(sentences, probs):\n",
        "  print(sent, prob)\n",
        "print(\"#########################\\n\")\n",
        "\n",
        "sentences, probs = generate_sentences(prefix=\"<BOS1> <BOS2> biden\", beam=10, sampler=sample_next_word)\n",
        "for sent, prob in zip(sentences, probs):\n",
        "  print(sent, prob)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<BOS1> <BOS2> trump eyes new unproven coronavirus treatment URL <EOS> 0.00021893147502903603\n",
            "<BOS1> <BOS2> trump eyes new unproven coronavirus cure URL <EOS> 0.0001719607222046247\n",
            "<BOS1> <BOS2> trump eyes new unproven virus cure promoted by mypillow ceo over unproven therapeutic URL <EOS> 9.773272077557522e-05\n",
            "<BOS1> <BOS2> trump eyes new unproven coronavirus therapeutic mypillow creator over unproven therapeutic URL <EOS> 8.212549111137046e-05\n",
            "<BOS1> <BOS2> trump eyes new unproven virus cure promoted by mypillow ceo over unproven therapeutic URL via @USER <EOS> 7.432226908194607e-06\n",
            "<BOS1> <BOS2> trump eyes new unproven virus cure promoted by mypillow ceo over unproven and dangerous <EOS> 5.61685494684627e-06\n",
            "<BOS1> <BOS2> trump eyes new unproven virus cure promoted by mypillow ceo over unproven and dangerous covid-19 treatment URL <EOS> 5.235550241426875e-06\n",
            "<BOS1> <BOS2> trump eyes new unproven virus cure promoted by ben carson and mypillow founder and ceo of mypillow URL <EOS> 2.1484173056680325e-06\n",
            "<BOS1> <BOS2> trump eyes new unproven virus cure promoted by ben carson and mypillow founder and ceo mike lindell a big deal <EOS> 1.4263568494891567e-08\n",
            "<BOS1> <BOS2> trump eyes new unproven virus cure promoted by ben carson and mypillow founder and ceo mike lindell a big deal and the pandemic <EOS> 4.890594996717161e-12\n",
            "#########################\n",
            "\n",
            "<BOS1> <BOS2> biden calls for a 30 bonus URL #cashgem #cashappfriday #stayathome <EOS> 0.0002495268686322749\n",
            "<BOS1> <BOS2> biden says all u.s. governors should mandate masks <EOS> 1.6894510541025754e-05\n",
            "<BOS1> <BOS2> biden says all u.s. governors question cost of a pandemic <EOS> 8.777606198953028e-07\n",
            "<BOS1> <BOS2> biden says all u.s. governors should mandate mandatory mask rule URL <EOS> 6.46094976762742e-07\n",
            "<BOS1> <BOS2> biden says all u.s. governors should mandate masks and social distancing <EOS> 4.6833316176693136e-07\n",
            "<BOS1> <BOS2> biden says all u.s. governors should mandate mandatory mask wearing and social distancing <EOS> 4.380454675651422e-08\n",
            "<BOS1> <BOS2> biden says all u.s. governors should mandate mandatory mask wearing and social distancing URL <EOS> 2.2277082512658355e-08\n",
            "<BOS1> <BOS2> biden says all u.s. governors should mandate mandatory mask wearing and social distancing and wearing masks <EOS> 1.166766912614153e-10\n",
            "<BOS1> <BOS2> biden says all u.s. governors should mandate mandatory mask wearing and social distancing and wearing a mask <EOS> 1.1355525098965152e-10\n",
            "<BOS1> <BOS2> biden says all u.s. governors should mandate mandatory mask wearing and social distancing and wearing a mask and social distancing and masks are not the same time <EOS> 2.3660719518591428e-20\n",
            "#########################\n",
            "\n",
            "<BOS1> <BOS2> trump 's pandemic response URL <EOS> 2.3321546760267828e-05\n",
            "<BOS1> <BOS2> trump suggests no more URL <EOS> 1.783043132055238e-05\n",
            "<BOS1> <BOS2> trump 's america failed the test URL <EOS> 1.3910086826662205e-06\n",
            "<BOS1> <BOS2> trump 's america failed the american people URL <EOS> 1.0651797035130728e-06\n",
            "<BOS1> <BOS2> trump 's america failed the #covid19 pandemic URL <EOS> 7.235784490295891e-07\n",
            "<BOS1> <BOS2> trump 's america failed the american era – rolling stone URL <EOS> 1.6705500613225425e-07\n",
            "<BOS1> <BOS2> trump 's america failed the american people during a pandemic <EOS> 2.6969768595862004e-08\n",
            "<BOS1> <BOS2> trump 's america failed the american era – rolling averages – 14th aug URL <EOS> 1.0946624333452558e-08\n",
            "<BOS1> <BOS2> trump 's america failed the american era – rolling averages – 10th aug URL <EOS> 5.473312166726279e-09\n",
            "<BOS1> <BOS2> trump 's america failed the american era – rolling averages – 14th aug URL via @USER <EOS> 1.248677954722147e-09\n",
            "#########################\n",
            "\n",
            "<BOS1> <BOS2> biden harris <EOS> 0.005673212681298934\n",
            "<BOS1> <BOS2> biden tells lin manuel miranda URL <EOS> 0.0010482180293501049\n",
            "<BOS1> <BOS2> biden tells lin manuel miranda latinos keeping us safe <EOS> 4.19287211740042e-05\n",
            "<BOS1> <BOS2> biden tells lin manuel miranda latinos keeping us safe URL <EOS> 3.975773935401521e-05\n",
            "<BOS1> <BOS2> biden tells lin manuel miranda latinos ‘ keeping us safe <EOS> 2.09643605870021e-05\n",
            "<BOS1> <BOS2> biden tells lin manuel miranda latinos ‘ keeping us safe URL <EOS> 1.9878869677007606e-05\n",
            "<BOS1> <BOS2> biden tells lin manuel miranda latinos ‘ keeping us safe from covid-19 #every80seconds <EOS> 1.7142943716549077e-06\n",
            "<BOS1> <BOS2> biden tells lin manuel miranda latinos ‘ keeping us safe from covid <EOS> 1.4082642579714798e-06\n",
            "<BOS1> <BOS2> biden tells lin manuel miranda latinos ‘ keeping us going during this pandemic <EOS> 9.341453280389532e-07\n",
            "<BOS1> <BOS2> biden tells lin manuel miranda latinos ‘ keeping us going during the pandemic is over <EOS> 1.1875921260783036e-08\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UShw7ULDcOwU"
      },
      "source": [
        "# 2. Semantic Parsing\n",
        "\n",
        "In this part, you are going to build your own virtual assistant! We will be developing two modules: an intent classifier and a slot filler."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dj2nWqZ9dZUs",
        "outputId": "2d5fab59-578f-41b3-cad6-4506f1fc140c"
      },
      "source": [
        "!ls \"/content/drive/My Drive/nl2ds/semantic-parser\"\n",
        "parser_files = \"/content/drive/My Drive/nl2ds/semantic-parser\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "test_answers.txt  test_questions.txt  train_questions_answers.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LbtOC6eecMNp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e00cfa51-43fa-4f0b-bd1f-658e08b93d71"
      },
      "source": [
        "import json\n",
        "\n",
        "train_data = []\n",
        "for line in open(f'{parser_files}/train_questions_answers.txt'):\n",
        "    train_data.append(json.loads(line))\n",
        "\n",
        "# print a few examples\n",
        "for i in range(5):\n",
        "    print(train_data[i])\n",
        "    print(\"-\"*80)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'question': 'Add an album to my Sylvia Plath playlist.', 'intent': 'AddToPlaylist', 'slots': {'music_item': 'album', 'playlist_owner': 'my', 'playlist': 'Sylvia Plath'}}\n",
            "--------------------------------------------------------------------------------\n",
            "{'question': 'add Diarios de Bicicleta to my la la playlist', 'intent': 'AddToPlaylist', 'slots': {'playlist': 'Diarios de Bicicleta', 'playlist_owner': 'my', 'entity_name': 'la la'}}\n",
            "--------------------------------------------------------------------------------\n",
            "{'question': 'book a table at a restaurant in Lucerne Valley that serves chicken nugget', 'intent': 'BookRestaurant', 'slots': {'restaurant_type': 'restaurant', 'city': 'Lucerne Valley', 'served_dish': 'chicken nugget'}}\n",
            "--------------------------------------------------------------------------------\n",
            "{'question': 'add iemand als jij to my playlist named In The Name Of Blues', 'intent': 'AddToPlaylist', 'slots': {'entity_name': 'iemand als jij', 'playlist_owner': 'my', 'playlist': 'In The Name Of Blues'}}\n",
            "--------------------------------------------------------------------------------\n",
            "{'question': 'What will the weather be in the current position on Dec. 23?', 'intent': 'GetWeather', 'slots': {'current_location': 'current position', 'timeRange': 'Dec. 23'}}\n",
            "--------------------------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BMV-NkkAb6X3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8a22b9fb-04a6-40dd-ccf6-30318f1b33b9"
      },
      "source": [
        "test_questions = []\n",
        "for line in open(f'{parser_files}/test_questions.txt'):\n",
        "    test_questions.append(json.loads(line))\n",
        "\n",
        "test_answers = []\n",
        "for line in open(f'{parser_files}/test_answers.txt'):\n",
        "    test_answers.append(json.loads(line))\n",
        "\n",
        "# print a few examples\n",
        "for i in range(5):\n",
        "    print(test_questions[i])\n",
        "    print(test_answers[i])\n",
        "    print(\"-\"*80)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Add an artist to Jukebox Boogie Rhythm & Blues\n",
            "{'intent': 'AddToPlaylist', 'slots': {'music_item': 'artist', 'playlist': 'Jukebox Boogie Rhythm & Blues'}}\n",
            "--------------------------------------------------------------------------------\n",
            "Will it be rainy at Sunrise in Ramey Saudi Arabia?\n",
            "{'intent': 'GetWeather', 'slots': {'condition_description': 'rainy', 'timeRange': 'Sunrise', 'city': 'Ramey', 'country': 'Saudi Arabia'}}\n",
            "--------------------------------------------------------------------------------\n",
            "Weather in two hours  in Uzbekistan\n",
            "{'intent': 'GetWeather', 'slots': {'timeRange': 'in two hours', 'country': 'Uzbekistan'}}\n",
            "--------------------------------------------------------------------------------\n",
            "Will there be a cloud in VI in 14 minutes ?\n",
            "{'intent': 'GetWeather', 'slots': {'condition_description': 'cloud', 'state': 'VI', 'timeRange': 'in 14 minutes'}}\n",
            "--------------------------------------------------------------------------------\n",
            "add nuba to my Metal Party playlist\n",
            "{'intent': 'AddToPlaylist', 'slots': {'entity_name': 'nuba', 'playlist_owner': 'my', 'playlist': 'Metal Party'}}\n",
            "--------------------------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hLUozTj613Tk"
      },
      "source": [
        "## Problem 2.1: Keyword-based intent classifier\n",
        "\n",
        "In this part, you will build a keyword-based intent classifier. For each intent, come up with a list of keywords that are important for that intent, and then classify a given question into an intent. If an input question matches multiple intents, pick the best one. If it does not match any keyword, return None.\n",
        "\n",
        "Caution: You are allowed to look at training questions and answers to come up with a set of keywords, but it is a bad practice to look at test answers. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fIOcz3lC4VqP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "13ed49d6-ed33-49b9-ff7e-20594fdff59d"
      },
      "source": [
        "# List of all intents\n",
        "intents = set()\n",
        "for example in train_data:\n",
        "    intents.add(example['intent'])\n",
        "print(intents)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'BookRestaurant', 'GetWeather', 'AddToPlaylist'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LmpLWuCO46NG"
      },
      "source": [
        "def predict_intent_using_keywords(question):\n",
        "  # Fill in your code here.\n",
        "  intent=''\n",
        "  q=question.lower().split()\n",
        "  if 'will' and 'be' and 'in' in q:\n",
        "    intent='GetWeather'\n",
        "  if 'be' in q:\n",
        "    intent='GetWeather'\n",
        "  if 'weather' in q:\n",
        "    intent='GetWeather'\n",
        "  if 'book' in q:\n",
        "    intent='BookRestaurant'\n",
        "  if 'add' in q:\n",
        "    intent='AddToPlaylist'\n",
        "  if 'playlist' in q:\n",
        "    intent='AddToPlaylist'\n",
        "  return intent"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BSNHoHR16jk9",
        "outputId": "a5daa1a9-4492-4c17-850c-5006c5f385e0"
      },
      "source": [
        "from collections import Counter\n",
        "\n",
        "'''Gives intent wise accuracy of your model'''\n",
        "def evaluate_intent_accuracy(prediction_function_name):\n",
        "  correct = Counter()\n",
        "  total = Counter()\n",
        "  for i in range(len(test_questions)):\n",
        "    q = test_questions[i]\n",
        "    gold_intent = test_answers[i]['intent']\n",
        "    if prediction_function_name(q) == gold_intent:\n",
        "      correct[gold_intent] += 1\n",
        "    total[gold_intent] += 1\n",
        "  for intent in intents:\n",
        "    print(intent, correct[intent]/total[intent], total[intent])\n",
        "    \n",
        "# Evaluating the intent classifier. \n",
        "# In our implementation, a simple keyword based classifier has achieved an accuracy of greater than 65 for each intent\n",
        "evaluate_intent_accuracy(predict_intent_using_keywords)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "BookRestaurant 0.86 100\n",
            "GetWeather 0.84 100\n",
            "AddToPlaylist 0.9 100\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RzV5NYJe-rbm"
      },
      "source": [
        "## Problem 2.2: Statistical intent classifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jizp2fxbb6X5"
      },
      "source": [
        "Now, let's build a statistical intent classifier. Instead of making use of keywords like what you did above, you will first extract features from a given input question. In order to build a feature representation for a given sentence, make use of word2vec embeddings of each word and take an average to represent the sentence. Then train a logistic regression. Feel free to use any libraries you like."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GPd4QZBhGOsm",
        "outputId": "9644b41b-5d20-46d0-a6c6-b95a696bf0f7"
      },
      "source": [
        "import nltk\n",
        "nltk.download('word2vec_sample')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package word2vec_sample to /root/nltk_data...\n",
            "[nltk_data]   Unzipping models/word2vec_sample.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8iys8Cb3An3x"
      },
      "source": [
        "from nltk.data import find\n",
        "import gensim\n",
        "\n",
        "word2vec_sample = str(find('models/word2vec_sample/pruned.word2vec.txt'))\n",
        "word2vec_model = gensim.models.KeyedVectors.load_word2vec_format(word2vec_sample, binary=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JrjgoaOIFM81"
      },
      "source": [
        "train_question=[d['question'] for d in train_data] #Extract questions and intents from the training dataset \n",
        "train_intent=[d['intent'] for d in train_data]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mpk3_JtMb6X6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1369317f-be6d-4db4-c952-75be7482a9d8"
      },
      "source": [
        "'''Trains a logistic regression model on the entire training data. For an input question (x), the model learns to predict an intent (Y).'''\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "import numpy as np\n",
        "def sentence_vector(sentence):\n",
        "    sentence = [word for word in sentence if word in word2vec_model.vocab] #Helper method to return representation for the sentence   \n",
        "    return np.mean(word2vec_model[sentence],axis=0)\n",
        "\n",
        "def train_logistic_regression_intent_classifier(questions,intents):\n",
        "    # Fill in your code here\n",
        "    avg_pred=[]\n",
        "    avg_intent=[]\n",
        "    for question in questions:\n",
        "      sent_em=sentence_vector(question)\n",
        "      avg_pred.append(sent_em)\n",
        "    X=np.array(avg_pred)\n",
        "    lr_model=LogisticRegression()\n",
        "    lr_model.fit(X, intents)\n",
        "\n",
        "\n",
        "    # Feel free to add more cells or functions if needed\n",
        "    return lr_model\n",
        "lr_model=train_logistic_regression_intent_classifier(train_question,train_intent)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_logistic.py:818: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "Please also refer to the documentation for alternative solver options:\n",
            "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
            "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q_1lHAw9b6X6"
      },
      "source": [
        "'''For an input question, the model predicts an intent'''\n",
        "def predict_intent_using_logistic_regression(question):\n",
        "    # Fill in your code here\n",
        "    test_list=[]\n",
        "    test_em=sentence_vector(question)\n",
        "    # Feel free to add more cells or functions if needed\n",
        "    return lr_model.predict(test_em.reshape(1,-1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yBwjBJoUb6X7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f66900df-877f-4293-b1aa-fcbdc40ee3ac"
      },
      "source": [
        "# Evaluate the intent classifier\n",
        "# Your intent classifier performance will be close to 100 if you have done a good job.\n",
        "evaluate_intent_accuracy(predict_intent_using_logistic_regression)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GetWeather 0.91 100\n",
            "BookRestaurant 0.93 100\n",
            "AddToPlaylist 0.9 100\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUnGNOHNXbSN"
      },
      "source": [
        "## Problem 2.3: Slot filling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ONXIMs6_b6X7"
      },
      "source": [
        "Build a slot filling model. We will just work with `AddToPlaylist` intent. Ignore other intents.\n",
        "\n",
        "Hint: No need to rely on machine learning here. You can use ideas like maximum string matching to identify which slots are active and what thier values are. This problem's solution is intentionally left underspecified."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xpg1x-qeb6X7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0868e1ca-7de8-4eee-f6f7-4ecb474293fb"
      },
      "source": [
        "# Let's stick to one target intent.\n",
        "target_intent = \"AddToPlaylist\"\n",
        "\n",
        "# This intent has the following slots\n",
        "target_intent_slot_names = set()\n",
        "for sample in train_data:\n",
        "    if sample['intent'] == target_intent:\n",
        "        for slot_name in sample['slots']:\n",
        "            target_intent_slot_names.add(slot_name)\n",
        "print(target_intent_slot_names)\n",
        "\n",
        "\n",
        "# Extract all the relevant questions of this target intent from the test examples.\n",
        "target_intent_questions = []\n",
        "for i, question in enumerate(test_questions):\n",
        "    if test_answers[i]['intent'] == target_intent:\n",
        "        target_intent_questions.append(question)\n",
        "print(target_intent_questions)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'playlist', 'playlist_owner', 'artist', 'entity_name', 'music_item'}\n",
            "['Add an artist to Jukebox Boogie Rhythm & Blues', 'add nuba to my Metal Party playlist', 'Add the album to the The Sweet Suite playlist.', 'Add a song to my playlist madden nfl 16', 'Can you put this song from Yutaka Ozaki onto my this is miles davis playlist?', 'Add give us rest to my 70s Smash Hits playlist.', 'Add this album to the Spanish Beat playlist', 'Add lofty fake anagram to the la mejor música de bso playlist.', 'Add the track to the drum & breaks playlist.', 'Add porter wagoner to the The Sleep Machine Waterscapes playlist.', 'Add Richard McNamara newest song to the Just Smile playlist', 'Add Recalled to Life to This Is Alejandro Fernández', 'Add the chris clark tune to my women of the blues playlist.', 'Put Jean Philippe Goncalves onto my running to rock 170 to 190 bpm.', 'add a song in my All Out 60s', 'Add The Famous Flower of Serving-Men to my evening acoustic playlist.', 'add tommy johnson to The MetalSucks Playlist', 'Can you add Larry Heard to my laundry playlist?', 'add this track to my Hands Up playlist', 'Add the album to the hot 50 playlist.', \"Add the song to the Mac 'N Cheese playlist.\", 'Add our little corner of the world music from gilmore girls to my The Funny Thing About Football Is playlist.', 'Add the album to the Might and Myth Power Metal playlist.', 'add this song by Too Poetic to my Piano Ballads playlist', 'Add the album to my Flow Español playlist.', 'add this artist to my spotlight on country 2016 playlist', 'Add this artist to Showstopper Being Mary Jane', 'Add the album to the Hipster Soul playlist.', 'Add Ted Heath to the road trip playlist.', 'For my fantastic workout can you add sara bareilles?', 'Add the 40 cal tune to the laundry playlist.', 'Add Roel van Velzen to my party of the century playlist.', \"Put Vandemataram Srinivas's track onto HipHop Hot 50.\", 'add Jonathan Sprout album to my this is miranda lambert playlist', 'To the travelling playlist, please add this David Gahan song.', \"Onto jerry's Classical Moments in Movies, please add the album.\", 'Add millie corretjer to the rhythm playlist', 'Add the track to the work playlist.', 'Add the artist to my dishwashing playlist.', 'can you add the current tune to my Calm before the storm playlist', 'Please add this track by Paul McGuigan to the Deep House playlist.', \"Add this tune to dorthy's 80's party playlist\", 'Add the artist to the political punks playlist.', 'add Beyond the Valley of 1984 in playlist Folk Music At The Gaslight Café', 'Add this album to Old School Death Metal', 'Put abe laboriel onto the Escapada playlist.', \"I'd like for you to add bobby brown to my enamorándose playlist.\", 'Add the artist mike to the Sexy as Folk playlist.', 'Add Andy Hunter to my Evening Commute playlist.', 'Add sarah slean to my playlist Mellowed Out Gaming', 'add track in my\\n playlist called Hands Up', 'I need to add baro ferret to the Urban Hits under my name', 'add ireland in the junior eurovision song contest 2015 to my Jazzy Dinner playlist', 'Add jo stafford music to the workout twerkout playlist', 'add this artist to my Electronic Bliss playlist', 'Can I put this tune onto my sin estrés playlist?', 'add the current track to my This Is Tchaikovsky playlist', 'Add martin barre to my punk unplugged playlist.', 'add tierney sutton to my Novedades Viernes Sudamérica playlist', 'Add Annesley Malewana album to playlist indietronic', 'Can you put the artist Giovanni Giacomo Gastoldi onto the Chill Out Music playlist?', 'add Highway Patrolman in my playlist this is al green', 'Add kent james to the disney soundtrack.', 'Add Brazilian Flag Anthem to Top 100 Alternative Tracks on Spotify', 'Add the song virales de siempre by the cary brothers to my gym playlist.', 'add the song to my We Everywhere playlist', 'Please add The image of you to my playlist Crate Diggers Anonymous', 'Add this artist to fairy tales playlist', 'I need to add to my Infinite Indie Folk list the works of Rahim Shah', \"I'd like to have this track onto my Classical Relaxations playlist.\", 'A Very Cellular Song needs to be added to my masters of metal playlist', 'Add a track to playlist Cena con Amigos', 'Add the artist Adam Deibert to my Perfect Concentration playlist.', 'add A Very Cellular Song to masters of metal playlist', 'add this tune to my Sleepify playlist', 'add Farhad Darya songs in virales de siempre', 'Add jack white to my playlist This Is Shakira', 'Can you add last of the ghetto astronauts to the playlist called black sabbath the dio years?', 'Add the album to my Perfect Concentration playlist.', 'add the artist Pete Murray to my relaxing playlist', 'add we have a theme song to my House Afterwork playlist', 'Add Michael Wittig music to Country Icon playlist', 'Add Toyan to my Epic Gaming playlist.', 'Add emilie autumn to my Nação Reggae playlist.', 'add digging now to my Young at Heart playlist', 'Add a derek watkins tune to This Is Johnny Cash', \"add abacab to beryl's party on fridays playlist\", 'Add the Matt Murphy tune to the Flow Español playlist.', \"I'd like to add jordan rudess onto the Divertido para niños playlist.\", 'Add the album to my Club Hits playlist.', \"Add a song to this is racionais mc's\", 'Put petar georgiev kalica onto the Old School Hip Hop playlist.', 'Add a track to Jazzy Dinner', \"I'd like for Kasey Chambers's tune to be an addition to my Chips and Salsa playlist.\", 'Add Jerry Calliste, Jr to my Te quiero playlist.', 'Add the boy george track to the Emo Forever playlist.', 'Please add some Pete Townshend to my playlist Fiesta Hits con Lali', 'Add a song to Indie Hipster', 'add Muzika za decu to my Crash Course playlist', 'put the artist onto Top Latin Alternative.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JIwJH7JPXgJz"
      },
      "source": [
        "target_intent_answers = []#Build a list for answers with intent \"AddToPlaylist\"\n",
        "for i, answer in enumerate(test_answers):\n",
        "    if test_answers[i]['intent'] == target_intent:\n",
        "        target_intent_answers.append(answer['slots'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yVEBIuBt-nVq"
      },
      "source": [
        "train_slots=[d['slots'] for d in train_data] #Retrieve slots from the training dataset \n",
        "slot_dict={} #Build a dictionary with slot names as keys and slot types as values \n",
        "for slot in train_slots:\n",
        "    key_list = list(slot.keys()) \n",
        "    val_list = list(slot.values())\n",
        "    for i in range(0,len(key_list)-1):\n",
        "      slot_dict[val_list[i]]=key_list[i]\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A7_ldSKob6X8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aa31730c-a1c9-4e68-c12e-e942577bfae5"
      },
      "source": [
        "def initialize_slots():\n",
        "    slots = {}\n",
        "    for slot_name in target_intent_slot_names:\n",
        "        slots[slot_name] = None\n",
        "    return slots\n",
        "\n",
        "def predict_slot_values(question):\n",
        "    slots = initialize_slots() \n",
        "    q=question.lower().split() #Tokenize the question\n",
        "    for word in q:\n",
        "      for slot_name in target_intent_slot_names:\n",
        "          if word in slot_dict.keys(): #Check if the tokens are in the dictionary\n",
        "            slots[slot_name]=word\n",
        "    return slots\n",
        "\n",
        "def evaluate_slot_prediction_recall(slot_prediction_function):\n",
        "    correct = Counter()\n",
        "    total = Counter() \n",
        "    # predict slots for each question\n",
        "    for i, question in enumerate(target_intent_questions):\n",
        "        i = test_questions.index(question) # This line is added after the assignment release\n",
        "        gold_slots = test_answers[i]['slots']\n",
        "        predicted_slots = slot_prediction_function(question)\n",
        "        for name in target_intent_slot_names:\n",
        "            if name in gold_slots:\n",
        "                total[name] += 1.0\n",
        "                if predicted_slots.get(name, None).lower() if predicted_slots.get(name, None) is not None else None == gold_slots.get(name).lower(): # This line is updated after the assignment release\n",
        "                    correct[name] += 1.0\n",
        "    for name in target_intent_slot_names:\n",
        "        print(f\"{name}: {correct[name] / total[name]}\")\n",
        "\n",
        "\n",
        "\n",
        "# Our reference implementation got these numbers. You can ask others on Slack what they got.\n",
        "# music_item 1.0\n",
        "# playlist 0.67\n",
        "# artist  0.021739130434782608\n",
        "# playlist_owner 0.9444444444444444\n",
        "# entity_name 0.1111111111111111\n",
        "print(\"Slot accuracy for your slot prediction model\")\n",
        "evaluate_slot_prediction_recall(predict_slot_values)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Slot accuracy for your slot prediction model\n",
            "playlist: 0.84\n",
            "playlist_owner: 0.9629629629629629\n",
            "artist: 0.782608695652174\n",
            "entity_name: 0.7222222222222222\n",
            "music_item: 0.9636363636363636\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lm4reFpdb6X8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0dceca8a-d9b7-4767-89e5-ab9b3efef1da"
      },
      "source": [
        "# Find a true positive prediction for each slot\n",
        "# Fill in your code below along with printing your prediction and gold answer\n",
        "break_num=0\n",
        "for i in range(0,len(target_intent_questions)-1):\n",
        "   predicted_slot={}\n",
        "   predicted_slot=predict_slot_values(target_intent_questions[i])\n",
        "   True_POS={}#Empty dict for output\n",
        "\n",
        "   for predicted_key in predicted_slot.keys():\n",
        "       if predicted_key in target_intent_answers[i].keys() and predicted_slot[predicted_key]==target_intent_answers[i][predicted_key]: #Condition for True Positive\n",
        "         True_POS[predicted_key]=predicted_slot[predicted_key]\n",
        "\n",
        "   if len(True_POS)!=0 and break_num<4:#Print 4 examples and break \n",
        "    break_num+=1\n",
        "    print(\"Question:\")\n",
        "    print(target_intent_questions[i])\n",
        "    print(\"Target:\") #Print target answers\n",
        "    print(target_intent_answers[i])\n",
        "    print(\"True Positive:\")\n",
        "    print(True_POS)\n",
        "    print('-'*80)\n",
        "  "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question:\n",
            "Add an artist to Jukebox Boogie Rhythm & Blues\n",
            "Target:\n",
            "{'music_item': 'artist', 'playlist': 'Jukebox Boogie Rhythm & Blues'}\n",
            "True Positive:\n",
            "{'music_item': 'artist'}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "add nuba to my Metal Party playlist\n",
            "Target:\n",
            "{'entity_name': 'nuba', 'playlist_owner': 'my', 'playlist': 'Metal Party'}\n",
            "True Positive:\n",
            "{'playlist_owner': 'my'}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "Add the album to the The Sweet Suite playlist.\n",
            "Target:\n",
            "{'music_item': 'album', 'playlist': 'The Sweet Suite'}\n",
            "True Positive:\n",
            "{'music_item': 'album'}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "Add a song to my playlist madden nfl 16\n",
            "Target:\n",
            "{'music_item': 'song', 'playlist_owner': 'my', 'playlist': 'madden nfl 16'}\n",
            "True Positive:\n",
            "{'playlist_owner': 'my'}\n",
            "--------------------------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "a=torch.rand(1,4)\n",
        "b=torch.rand(4,4)\n",
        "cos = nn.CosineSimilarity(dim=1)\n",
        "output = cos(a, b)\n",
        "print(a)\n",
        "print(b)\n",
        "print(output)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2kWx8s-JEUs0",
        "outputId": "c6df93bf-2f28-405a-db2a-2784e73290cc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[0.0523, 0.5786, 0.2013, 0.6792]])\n",
            "tensor([[0.3773, 0.9794, 0.6747, 0.2990],\n",
            "        [0.6708, 0.5720, 0.3575, 0.5668],\n",
            "        [0.8677, 0.2762, 0.2937, 0.1248],\n",
            "        [0.4726, 0.9698, 0.0773, 0.3016]])\n",
            "tensor([0.7872, 0.8112, 0.3949, 0.7838])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TUo4NblMb6X8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "71d49e92-a62e-4e6a-b38a-cfd5f2a7842d"
      },
      "source": [
        "# Find a false positive prediction for each slot\n",
        "# Fill in your code below along with print statement\n",
        "break_num=0\n",
        "false_positive={}\n",
        "for i in range(0,len(target_intent_questions)-1):\n",
        "   predicted_slot={}\n",
        "   predicted_slot=predict_slot_values(target_intent_questions[i])\n",
        "   False_POS={} #Empty dict to save output\n",
        "   for predicted_key in predicted_slot.keys():\n",
        "       if predicted_slot[predicted_key]!=None and predicted_key not in target_intent_answers[i].keys():\n",
        "         False_POS[predicted_key]=predicted_slot[predicted_key]\n",
        "   if len(False_POS)!=0 and break_num<4:\n",
        "    break_num+=1\n",
        "    print(\"Question:\")\n",
        "    print(target_intent_questions[i])\n",
        "    print(\"Target:\")\n",
        "    print(target_intent_answers[i])\n",
        "    print(\"False Positive:\")\n",
        "    print(False_POS)\n",
        "    print('-'*80)\n",
        "   "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question:\n",
            "Add an artist to Jukebox Boogie Rhythm & Blues\n",
            "Target:\n",
            "{'music_item': 'artist', 'playlist': 'Jukebox Boogie Rhythm & Blues'}\n",
            "False Positive:\n",
            "{'playlist_owner': 'artist', 'artist': 'artist', 'entity_name': 'artist'}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "add nuba to my Metal Party playlist\n",
            "Target:\n",
            "{'entity_name': 'nuba', 'playlist_owner': 'my', 'playlist': 'Metal Party'}\n",
            "False Positive:\n",
            "{'artist': 'my', 'music_item': 'my'}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "Add the album to the The Sweet Suite playlist.\n",
            "Target:\n",
            "{'music_item': 'album', 'playlist': 'The Sweet Suite'}\n",
            "False Positive:\n",
            "{'playlist_owner': 'album', 'artist': 'album', 'entity_name': 'album'}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "Add a song to my playlist madden nfl 16\n",
            "Target:\n",
            "{'music_item': 'song', 'playlist_owner': 'my', 'playlist': 'madden nfl 16'}\n",
            "False Positive:\n",
            "{'artist': 'my', 'entity_name': 'my'}\n",
            "--------------------------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "v1=[0,1,0,0,0]"
      ],
      "metadata": {
        "id": "saU_4useRhmw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yQjb1-TCb6X8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "40faf5e4-740a-4a6b-a2e5-cff97f4316c3"
      },
      "source": [
        "# Find a true negative prediction for each slot\n",
        "# Fill in your code below along with a print statement\n",
        "break_num=0\n",
        "for i in range(0,len(target_intent_questions)-1):\n",
        "   predicted_slot={}\n",
        "   predicted_slot=predict_slot_values(target_intent_questions[i])\n",
        "   True_NEG={}\n",
        "   target_slot={}\n",
        "   for predicted_key in predicted_slot.keys():\n",
        "       if predicted_slot[predicted_key]==None and predicted_key not in target_intent_answers[i].keys():\n",
        "         True_NEG[predicted_key]=predicted_slot[predicted_key]\n",
        "   if len(True_NEG)!=0 and break_num<4:\n",
        "    break_num+=1\n",
        "    print(\"Question:\")\n",
        "    print(target_intent_questions[i])\n",
        "    print(\"Target:\")\n",
        "    print(target_intent_answers[i])\n",
        "    print(\"True Negative:\")\n",
        "    print(True_NEG)\n",
        "    print('-'*80)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question:\n",
            "Add lofty fake anagram to the la mejor música de bso playlist.\n",
            "Target:\n",
            "{'entity_name': 'lofty fake anagram', 'playlist': 'la mejor música de bso'}\n",
            "True Negative:\n",
            "{'playlist_owner': None, 'artist': None, 'music_item': None}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "Add porter wagoner to the The Sleep Machine Waterscapes playlist.\n",
            "Target:\n",
            "{'artist': 'porter wagoner', 'playlist': 'The Sleep Machine Waterscapes'}\n",
            "True Negative:\n",
            "{'playlist_owner': None, 'entity_name': None, 'music_item': None}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "Add Recalled to Life to This Is Alejandro Fernández\n",
            "Target:\n",
            "{'entity_name': 'Recalled to Life', 'playlist': 'This Is Alejandro Fernández'}\n",
            "True Negative:\n",
            "{'playlist_owner': None, 'artist': None, 'music_item': None}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "add tommy johnson to The MetalSucks Playlist\n",
            "Target:\n",
            "{'artist': 'tommy johnson', 'playlist': 'The MetalSucks Playlist'}\n",
            "True Negative:\n",
            "{'playlist_owner': None, 'entity_name': None, 'music_item': None}\n",
            "--------------------------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UJHTfEMqb6X8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f95b5220-3184-44f6-cdbb-8cfa9317f677"
      },
      "source": [
        "# Find a false negative prediction for each slot\n",
        "# Fill in your code below along with a print statement\n",
        "break_num=0\n",
        "for i in range(0,len(target_intent_questions)-1):\n",
        "   predicted_slot={}\n",
        "   predicted_slot=predict_slot_values(target_intent_questions[i])\n",
        "   False_NEG={}\n",
        "   target_slot={}\n",
        "   for predicted_key in predicted_slot.keys():\n",
        "       if predicted_slot[predicted_key]==None and predicted_key in target_intent_answers[i].keys():\n",
        "         False_NEG[predicted_key]=None\n",
        "   if len(False_NEG)!=0 and break_num<4:\n",
        "    break_num+=1\n",
        "    print(\"Question:\")\n",
        "    print(target_intent_questions[i])\n",
        "    print(\"Target:\")\n",
        "    print(target_intent_answers[i])\n",
        "    print(\"False Negative:\")\n",
        "    print(False_NEG)\n",
        "    print('-'*80)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question:\n",
            "Add lofty fake anagram to the la mejor música de bso playlist.\n",
            "Target:\n",
            "{'entity_name': 'lofty fake anagram', 'playlist': 'la mejor música de bso'}\n",
            "False Negative:\n",
            "{'playlist': None, 'entity_name': None}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "Add porter wagoner to the The Sleep Machine Waterscapes playlist.\n",
            "Target:\n",
            "{'artist': 'porter wagoner', 'playlist': 'The Sleep Machine Waterscapes'}\n",
            "False Negative:\n",
            "{'playlist': None, 'artist': None}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "Add Recalled to Life to This Is Alejandro Fernández\n",
            "Target:\n",
            "{'entity_name': 'Recalled to Life', 'playlist': 'This Is Alejandro Fernández'}\n",
            "False Negative:\n",
            "{'playlist': None, 'entity_name': None}\n",
            "--------------------------------------------------------------------------------\n",
            "Question:\n",
            "add tommy johnson to The MetalSucks Playlist\n",
            "Target:\n",
            "{'artist': 'tommy johnson', 'playlist': 'The MetalSucks Playlist'}\n",
            "False Negative:\n",
            "{'playlist': None, 'artist': None}\n",
            "--------------------------------------------------------------------------------\n"
          ]
        }
      ]
    }
  ]
}