WK4_Natural_Language_Processing_Task.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount(\"/content/drive\")"
      ],
      "metadata": {
        "id": "zIAOGqAWqnNX",
        "outputId": "1d4924c5-e7ea-4044-e3f6-4325ad3b6bd2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install GitPython\n",
        "from git import Repo"
      ],
      "metadata": {
        "id": "qygF-2H8q6z9",
        "outputId": "9e990bd9-567a-4c3a-9945-1c48c15fb011",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting GitPython\n",
            "  Downloading GitPython-3.1.31-py3-none-any.whl (184 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m184.3/184.3 KB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gitdb<5,>=4.0.1\n",
            "  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 KB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting smmap<6,>=3.0.1\n",
            "  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n",
            "Installing collected packages: smmap, gitdb, GitPython\n",
            "Successfully installed GitPython-3.1.31 gitdb-4.0.10 smmap-5.0.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# the http is the clone link from github\n",
        "#while repo_local_dir is the directory we will store the cloned files\n",
        "https =\"https://github.com/Antony-gitau/data-science-in-healthcare-and-precision-medicine.git\"\n",
        "repo_local_dir = \"repo\""
      ],
      "metadata": {
        "id": "Xu0idtzdq_Mf"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "Repo.clone_from(https, repo_local_dir)"
      ],
      "metadata": {
        "id": "aMdWsoqArGxk",
        "outputId": "edbc9c72-187b-4feb-a628-b5e0db189408",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<git.repo.base.Repo '/content/repo/.git'>"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0e8lzHo2qiEv"
      },
      "source": [
        "# Natural Language Processing Task"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sqhtj-3-qiEy"
      },
      "source": [
        "In this programming task, we'll learn how to do some basic natural processing tasks, e.g. tokenisation, stemming and named entity recognition. We'll get to work with two fictitious clinical pieces of text: a biopsy report and a medical note."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CFrIOqVIqiEz"
      },
      "source": [
        "## Part 1: Importing packages needed"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pRjYfndtqiEz"
      },
      "source": [
        "### Importing NLTK"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AmNSjjMwqiEz"
      },
      "source": [
        "*NLTK* (Natural Language Toolkit) is a very popular suite of libraries and other resources for natural language processing. Let's tell Python that we're going to be using NLTK, with the use of the import command."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "TnrCoEGVqiE0"
      },
      "outputs": [],
      "source": [
        "import nltk"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_O9z6kgjqiE1"
      },
      "source": [
        "We will also import a few more NLTK utilities for tokenisation, stemming, part of speech tagging and named entity recognition."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "eBDxQWI_qiE2",
        "outputId": "1cd0d728-3620-4276-f316-1c2c057de557",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "[nltk_data] Downloading package maxent_ne_chunker to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping chunkers/maxent_ne_chunker.zip.\n",
            "[nltk_data] Downloading package words to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/words.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "from nltk.corpus import stopwords\n",
        "nltk.download('stopwords')\n",
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('maxent_ne_chunker')\n",
        "nltk.download('words')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EkPudfK8qiE2"
      },
      "source": [
        "### Importing docx2txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Lxac9szqiE2"
      },
      "source": [
        "*docx2txt* is a Python-based utility which converts text within docx files into plain text. We'll import this in order to be able to work with the two clinical docx files."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! pip install docx2txt"
      ],
      "metadata": {
        "id": "pZZK6SeyrzPo",
        "outputId": "d83a0cfc-2f9c-4973-b0fa-ef56a17d1722",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting docx2txt\n",
            "  Downloading docx2txt-0.8.tar.gz (2.8 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: docx2txt\n",
            "  Building wheel for docx2txt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for docx2txt: filename=docx2txt-0.8-py3-none-any.whl size=3980 sha256=5af4e9f6d70db566740be1413af72c75159aa657310c14e0ff4cd31f24faf14a\n",
            "  Stored in directory: /root/.cache/pip/wheels/40/75/01/e6c444034338bde9c7947d3467807f889123465c2371e77418\n",
            "Successfully built docx2txt\n",
            "Installing collected packages: docx2txt\n",
            "Successfully installed docx2txt-0.8\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "pagVnSM7qiE3"
      },
      "outputs": [],
      "source": [
        "import docx2txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yem2XZe6qiE3"
      },
      "source": [
        "<br>\n",
        "## Part 2: Processing a Biopsy Report"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sHWQkqEHqiE3"
      },
      "source": [
        "### Loading the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DVarXp2aqiE4"
      },
      "source": [
        "The Biopsy Report is a fictitious example for the purpose of this programming task. It contains biopsy details about a mastecotmy specimen from a patient.\n",
        "\n",
        "The Biopsy Report is a Microsoft Word document (in a docx file format). In order for it to be processed by the NLTK package, the document requires to be converted into plain text.  In this case, we are using the *process* method from the docx2txt Python package to do this. We are calling the resulting plain text \"text\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "collapsed": true,
        "id": "vnfXoCbBqiE4"
      },
      "outputs": [],
      "source": [
        "text = docx2txt.process('/content/repo/Biopsy_Report.docx')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32z2-gYzqiE5"
      },
      "source": [
        "By running the code in the following cell, you'll see that text is a string."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "6uiIsCFaqiE5",
        "outputId": "3d7f4920-68d3-4027-c8da-cdcccdfe8972",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "str"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "type(text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8zX6W5d0qiE5"
      },
      "source": [
        "Run the code in the following cell to get the first 160 characters in *text*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "cy8NLiBfqiE5",
        "outputId": "6da5c241-46f3-4614-ba6c-e9fa33472ff1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Biopsy Report\\n\\n\\n\\nName:  Mrs Jane Black\\t\\t\\t\\t\\tD.O.B.: 01-01-58\\n\\nHospital:  RIE\\t\\t\\t\\t\\t\\tPAS No:  12345\\t\\tWard:  1\\n\\nClinician:  A. Smith\\t\\t\\t\\t\\tSpeciality:  General Surgery'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "text[:160]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6MRpWmp1qiE5"
      },
      "source": [
        "You can recognise several words in this text, e.g. \"Biopsy Report\" and \"General Surgery\". Note that there are some special characters too:\n",
        "- \\n corresponds to new line\n",
        "- \\t corresponds to tab space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AMZ9iSFdqiE5"
      },
      "source": [
        "### Tokenisation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t4mLGDE-qiE6"
      },
      "source": [
        "Tokenisation breaks large strings into smaller chunks. Let's use NLTK's *word_tokenize* method to split the *text* string into words and punctuation. We'll call the resulting list of words and punctuation *tokens*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "WxA36zKMqiE6"
      },
      "outputs": [],
      "source": [
        "tokens = nltk.word_tokenize(text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oQU-RzqxqiE6"
      },
      "source": [
        "Run the code in the following cell to get the first 10 elements of *tokens*. As you can see, there are both word- and punctuation-based tokens."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "Tp_PX8AOqiE6",
        "outputId": "a0ad7a00-873b-491a-8814-8f2563aa9fa1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Biopsy', 'Report', 'Name', ':', 'Mrs', 'Jane', 'Black', 'D.O.B', '.', ':']"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "tokens[:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A44wZlJgqiE6"
      },
      "source": [
        "#### Removing stop words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JTZn7BaMqiE6"
      },
      "source": [
        "Now we want to remove any stop words (i.e. any commonly used words, such as \"and\") from the list of tokens. We'll call the resulting list of tokens *clean_tokens*. <br><br>\n",
        "Note that you don't need to understand the code in the following cell. You can simply reuse it in the future as a template, where you replace *tokens* with the name of your list of tokens.\n",
        "(But if you're really curious, this is what the following code does: It first creates a copy of *tokens* called *clean_tokens*. And then for each token in the list of tokens, if it's in the list of stop words in English, then it is removed.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "collapsed": true,
        "id": "CT243wy5qiE7"
      },
      "outputs": [],
      "source": [
        "clean_tokens = tokens[:]\n",
        "for token in tokens:\n",
        "    if token in stopwords.words('english'):\n",
        "        clean_tokens.remove(token)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "for loops are not idea for iterating over a list while deleting tokens. Since deltions alters the index of remaining tokens which could cause lead to skipping of some tokens.\n",
        "\n",
        "So we implement a list comprehension"
      ],
      "metadata": {
        "id": "AEk-k3ZMt0M8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "clean_tokens2 = [token for token in tokens if token not in stopwords.words('english')]\n"
      ],
      "metadata": {
        "id": "xo8XqJZyuLQj"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Dy1zLjjqiE7"
      },
      "source": [
        "Let's print out the number of tokens (with and without stop words)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "IgFXHaEGqiE7",
        "outputId": "0d0751bc-3237-4842-c7df-ef1efbc52c13",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of tokens including stop words:   170\n",
            "Number of tokens excluding stop words:   165\n",
            "without stop words using list comprehension 165\n"
          ]
        }
      ],
      "source": [
        "print(\"Number of tokens including stop words:  \",len(tokens))\n",
        "print(\"Number of tokens excluding stop words:  \",len(clean_tokens))\n",
        "print(\"without stop words using list comprehension\", len(clean_tokens2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j8PP-JyJqiE8"
      },
      "source": [
        "As you can see, there are not that many stop words in the Biopsy Report. However, removing stop words may be really useful when working with documents containing many pages of text.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rSsLVDWTqiE8"
      },
      "source": [
        "#### Frequency distributions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "moUcp8lfqiE8"
      },
      "source": [
        "Next, we want to investigate the frequency of certain words in the Biopsy Report. We start by generating the frequency distribution of all tokens (including stop words) with the use of NLTK's FreqDist. Let's call this *freq*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "collapsed": true,
        "id": "rIz0gv2LqiE8"
      },
      "outputs": [],
      "source": [
        "freq = nltk.FreqDist(tokens)\n",
        "freq_without_stopwords = nltk.FreqDist(clean_tokens2)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(freq)\n",
        "print(freq_without_stopwords)\n",
        "freq_without_stopwords"
      ],
      "metadata": {
        "id": "_XOyLHx1vBdP",
        "outputId": "37b9bcd3-304d-42ca-9d03-efe6f1fb83a7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<FreqDist with 126 samples and 170 outcomes>\n",
            "<FreqDist with 122 samples and 165 outcomes>\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "FreqDist({':': 12, 'Yes': 4, 'present': 4, 'Report': 3, 'No': 3, 'Present': 3, 'Specimen': 3, 'LESIONS': 3, 'Date': 2, 'Mastectomy': 2, ...})"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eh1qzJ5UqiE9"
      },
      "source": [
        "Run the code in the following cell to get the 10 most frequently encountered tokens and their frequencies."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "gbxHbRtgqiE9",
        "outputId": "75cd7ca8-478c-4788-d60d-fb2231807adb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(':', 12),\n",
              " ('Yes', 4),\n",
              " ('present', 4),\n",
              " ('Report', 3),\n",
              " ('No', 3),\n",
              " ('Present', 3),\n",
              " ('Specimen', 3),\n",
              " ('LESIONS', 3),\n",
              " ('Date', 2),\n",
              " ('Mastectomy', 2)]"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ],
      "source": [
        "freq.most_common(10)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "freq_without_stopwords.most_common(10)"
      ],
      "metadata": {
        "id": "higRSDNKvfwf",
        "outputId": "f19eacde-3442-4ea0-82c0-5dfe5fcd4cdf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(':', 12),\n",
              " ('Yes', 4),\n",
              " ('present', 4),\n",
              " ('Report', 3),\n",
              " ('No', 3),\n",
              " ('Present', 3),\n",
              " ('Specimen', 3),\n",
              " ('LESIONS', 3),\n",
              " ('Date', 2),\n",
              " ('Mastectomy', 2)]"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vw7u6t1XqiE9"
      },
      "source": [
        "Taking \"lesion\" and \"lesions\" as an example, we can see how many times they appear in the document. Note that we're distinguishing beween different versions of the word, such as upper case and lower case. These are not the only possible variations, they're just the ones we're interested in right now."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "scrolled": true,
        "id": "fMLUljXZqiE-",
        "outputId": "b0fcdba0-f1c0-4c56-ff14-594fd3931d6e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Frequency of lesion:   1\n",
            "Frequency of lesions:  2\n",
            "Frequency of LESION:   0\n",
            "Frequency of LESIONS:  3\n"
          ]
        }
      ],
      "source": [
        "print(\"Frequency of lesion:  \", freq[\"lesion\"])\n",
        "print(\"Frequency of lesions: \", freq[\"lesions\"])\n",
        "print(\"Frequency of LESION:  \", freq[\"LESION\"])\n",
        "print(\"Frequency of LESIONS: \", freq[\"LESIONS\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "exglckoDqiE-"
      },
      "source": [
        "#### Lower case vs. upper case text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ChbAetg5qiE_"
      },
      "source": [
        "It can be useful to disregard upper case text and lower case text.  Therefore, we can convert upper case text to lower case text. We can do this with the use of Python's *lower()* method. <br>\n",
        "\n",
        "Let's apply the *lower()* method to each element in *tokens*, so that all text is counted as lower case text. We'll call this new list of lower case tokens *lowercase_tokens* (but you could call it anything you want)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "collapsed": true,
        "id": "jBS5GC4JqiE_"
      },
      "outputs": [],
      "source": [
        "lowercase_tokens = [t.lower() for t in tokens]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Sd5xOKZqiE_"
      },
      "source": [
        "Run the code in the following cell to get the first 10 elements of *lowercase_tokens*. As you can see, this is all lower case."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "cdCmbumOqiFA",
        "outputId": "d63682dc-21dd-4727-c6ff-cd91270b5951",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['biopsy', 'report', 'name', ':', 'mrs', 'jane', 'black', 'd.o.b', '.', ':']"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ],
      "source": [
        "lowercase_tokens[:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AbZ4oIlNqiFA"
      },
      "source": [
        "Next, we'll generate the frequency distribution of *lowercase_tokens*, and we'll call it *lowercase_freq*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "collapsed": true,
        "id": "ao-9vvSUqiFA"
      },
      "outputs": [],
      "source": [
        "lowercase_freq = nltk.FreqDist(lowercase_tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JlQKMrltqiFA"
      },
      "source": [
        "Now, let's print out the frequency of lesion(s)/LESION(S) as an example.   \n",
        "Observe that the frequencies for LESION and LESIONS are now 0.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "DZPMvFrnqiFB",
        "outputId": "0b8808f9-bc19-45f4-e5ac-f0c488b182fd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Frequency of lesion:   1\n",
            "Frequency of lesions:  5\n",
            "Frequency of LESION:   0\n",
            "Frequency of LESIONS:  0\n"
          ]
        }
      ],
      "source": [
        "print(\"Frequency of lesion:  \", lowercase_freq[\"lesion\"])\n",
        "print(\"Frequency of lesions: \", lowercase_freq[\"lesions\"])\n",
        "print(\"Frequency of LESION:  \", lowercase_freq[\"LESION\"])\n",
        "print(\"Frequency of LESIONS: \", lowercase_freq[\"LESIONS\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "At2CSGPiqiFB"
      },
      "source": [
        "### Stemming"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FkgL-cDHqiFB"
      },
      "source": [
        "Stemming is the process of reducing a word to its stem.\n",
        "\n",
        "NLTK includes several off-the-shelf stemmers. In this task, we'll be using the Porter stemmer, which implements the Porter stemming algorithm. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "collapsed": true,
        "id": "87fOj2gqqiFB"
      },
      "outputs": [],
      "source": [
        "stemmer = nltk.PorterStemmer()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nfoq0XxyqiFC"
      },
      "source": [
        "Run the code in the following cell to get the stems of all tokens in the Biopsy Report.<br>\n",
        "Note that we're working with the converted lower case tokens. So all that we do here is apply *stem* on each token in *lowercase_tokens*. The resulting list of stemmed tokens (i.e. tokens where any affixes have been removed) is called *stem_tokens*. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "collapsed": true,
        "id": "B0Ok2GEeqiFC"
      },
      "outputs": [],
      "source": [
        "stem_tokens = lowercase_tokens\n",
        "stem_tokens[:] = [stemmer.stem(lt) for lt in lowercase_tokens]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4b5zmtSTqiFC"
      },
      "source": [
        "Next, we'll generate the frequency distribution of *stem_tokens*, and we'll call it *stem_freq*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "collapsed": true,
        "id": "TbUME-iHqiFC"
      },
      "outputs": [],
      "source": [
        "stem_freq = nltk.FreqDist(stem_tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a03wMe0VqiFD"
      },
      "source": [
        "Now, lets print out the freqency of lesion(s)/LESION(S) as an example.  \n",
        "Note that the frequency of \"lesions\" is now 0, while the frequency of \"lesion\" is 6.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "ZJp7-7RMqiFD",
        "outputId": "26c94705-74d8-4f10-8862-4446170532f0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Frequency of lesion:   6\n",
            "Frequency of lesions:  0\n",
            "Frequency of LESION:   0\n",
            "Frequency of LESIONS:  0\n"
          ]
        }
      ],
      "source": [
        "print(\"Frequency of lesion:  \", stem_freq[\"lesion\"])\n",
        "print(\"Frequency of lesions: \", stem_freq[\"lesions\"])\n",
        "print(\"Frequency of LESION:  \", stem_freq[\"LESION\"])\n",
        "print(\"Frequency of LESIONS: \", stem_freq[\"LESIONS\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zNS7gCkaqiFD"
      },
      "source": [
        "<br>\n",
        "## Part 3: Processing a Medical Note"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2dA813WaqiFE"
      },
      "source": [
        "### Loading the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H76bSpdkqiFE"
      },
      "source": [
        "The Medical Note is another made-up example for the purpose of this programming task. It contains information about a patient with low back pain.\n",
        "\n",
        "Just like the Biopsy Report, the Medical Note is a Microsoft Word document (in a docx file format), so we'll use the *process* method from the docx2txt Python package to convert it into plain text. The resulting plain text is called \"content\". "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "collapsed": true,
        "id": "e5fwaWW5qiFE"
      },
      "outputs": [],
      "source": [
        "content = docx2txt.process('/content/repo/Medical_Note.docx')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bxCH08b5qiFE"
      },
      "source": [
        "Run the code in the following cell to get the first 160 characters in *content*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "_lIWSayRqiFE",
        "outputId": "9a524bdf-1612-4012-e647-9e9879be6d60",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'30-year-old Caucasian male presents with low back pain for 3 days.  \\n\\n\\n\\nPatient was gardening when pain started in the low back on the left side.  Pain started '"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 37
        }
      ],
      "source": [
        "content[:160]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p61YpCEtqiFF"
      },
      "source": [
        "### Tokenisation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fvy3s_5qqiFF"
      },
      "source": [
        "We'll first tokenise the Medical Note content into sentences. We'll do this with the use of NLTK's *sent_tokenize* method and we'll call the resulting list of sentences *sents*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "collapsed": true,
        "id": "-FgzIXodqiFF"
      },
      "outputs": [],
      "source": [
        "sents = nltk.sent_tokenize(content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hU7QnWi5qiFF"
      },
      "source": [
        "Run the code in the following cell to get the first 4 elements of *sents*, i.e. the first 4 sentences."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "1VnbBUjwqiFF",
        "outputId": "37222613-af9f-40fd-e814-bf6d3c540645",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['30-year-old Caucasian male presents with low back pain for 3 days.',\n",
              " 'Patient was gardening when pain started in the low back on the left side.',\n",
              " 'Pain started at a 9/10 and now is a 7/10.',\n",
              " 'Pain described as tight and constant.']"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ],
      "source": [
        "sents[:4]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lGkCyj3JqiFG"
      },
      "source": [
        "We next want to further tokenise the second sentence into words and punctuation. Just like with the Biopsy Report, we'll use NLTK's *word_tokenize* method. <br>\n",
        "Remember that lists in Python are 0-indexed, so the second sentence has index 1 in *sents*. <br> \n",
        "We'll call the resulting list of words and punctuation *medical_tokens*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "collapsed": true,
        "id": "eF17C9nRqiFG"
      },
      "outputs": [],
      "source": [
        "medical_tokens = nltk.word_tokenize(sents[1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_XvMNFZVqiFG"
      },
      "source": [
        "Let's get the entire *medical_tokens* list."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "scrolled": true,
        "id": "z0xq8k3JqiFG",
        "outputId": "1b6664f6-1a38-459c-f79b-6587de51f77f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Patient',\n",
              " 'was',\n",
              " 'gardening',\n",
              " 'when',\n",
              " 'pain',\n",
              " 'started',\n",
              " 'in',\n",
              " 'the',\n",
              " 'low',\n",
              " 'back',\n",
              " 'on',\n",
              " 'the',\n",
              " 'left',\n",
              " 'side',\n",
              " '.']"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ],
      "source": [
        "medical_tokens"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UEq3qHAmqiFG"
      },
      "source": [
        "### Part-of-Speech Tagging"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n3VuPf0PqiFH"
      },
      "source": [
        "A Part-of-speech tagger (or POS-tagger) processes a sequence of words and attaches a part of speech tag to each word.  \n",
        "\n",
        "Run the code in the following cell to use NLTK's *pos_tag()* to attach a part of speech tag to each token in the second sentence. The resulting list is called *tagged*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "id": "sO8Br1pgqiFH"
      },
      "outputs": [],
      "source": [
        "tagged = nltk.pos_tag(medical_tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sTwR-qgUqiFH"
      },
      "source": [
        "Now, let's ask for *tagged*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "id": "gH6qbjWpqiFH",
        "outputId": "c9168ab9-fc22-4009-e186-0f76bc887d49",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Patient', 'NN'),\n",
              " ('was', 'VBD'),\n",
              " ('gardening', 'VBG'),\n",
              " ('when', 'WRB'),\n",
              " ('pain', 'NN'),\n",
              " ('started', 'VBD'),\n",
              " ('in', 'IN'),\n",
              " ('the', 'DT'),\n",
              " ('low', 'JJ'),\n",
              " ('back', 'RB'),\n",
              " ('on', 'IN'),\n",
              " ('the', 'DT'),\n",
              " ('left', 'JJ'),\n",
              " ('side', 'NN'),\n",
              " ('.', '.')]"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ],
      "source": [
        "tagged"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uy2Bv4cQqiFI"
      },
      "source": [
        "Here you can see many different POS tags, such as \"JJ\" and \"NN\". \n",
        "\n",
        "The meaning of commonly used POS tags is provided below:\n",
        "- NN: noun \n",
        "- JJ: adjective\n",
        "- DT: determiner\n",
        "- VB: verb in base form\n",
        "- VBD: verb in past tense\n",
        "- VBG: verb in gerund/present participle\n",
        "- RB: adverb\n",
        "- WRB wh-abverb (e.g. where)\n",
        "- IN: preposition/subordinating conjunction\n",
        "- CC: coordinating conjunction (e.g. and)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W9h6MO1CqiFI"
      },
      "source": [
        "### Named Entity Recognition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2hqqpQw4qiFI"
      },
      "source": [
        "Now that we have the parts of speech, we can try named entity recognition, which is concerned with the task of finding entities in text and classifying them as persons, locations, dates, and so on.<br>\n",
        "\n",
        "NLTK provides a classifier that has already been trained to recognise named entities, and which can be accessed with the *nltk.ne_chunk()* function. We'll apply it to *tagged* and call the resulting structure *entities*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "id": "Ly4wVGRtqiFJ"
      },
      "outputs": [],
      "source": [
        "entities = nltk.ne_chunk(tagged)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kB5cLWDIqiFJ"
      },
      "source": [
        "Let's print out *entities*. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "id": "-bZILaw_qiFJ",
        "outputId": "b88912df-b51d-4f6f-d546-9a04299f6414",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(S\n",
            "  (PERSON Patient/NN)\n",
            "  was/VBD\n",
            "  gardening/VBG\n",
            "  when/WRB\n",
            "  pain/NN\n",
            "  started/VBD\n",
            "  in/IN\n",
            "  the/DT\n",
            "  low/JJ\n",
            "  back/RB\n",
            "  on/IN\n",
            "  the/DT\n",
            "  left/JJ\n",
            "  side/NN\n",
            "  ./.)\n"
          ]
        }
      ],
      "source": [
        "print(entities)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Okj8cMOmqiFK"
      },
      "source": [
        "As we can see, \"Patient\" has been correctly classified as a Person."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VRq-f-iJqiFK"
      },
      "source": [
        "<br>\n",
        "## Part 4: Keep practising"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I6MjtpzCqiFK"
      },
      "source": [
        "### Quiz 4 - Question 5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ap0ofW7rqiFK"
      },
      "source": [
        "Make changes to the code below to find out how many times the word ‘malignant’ appears in the Biopsy Report.  This should include upper and lower case variations of this word – think about this carefully!  <br><br>\n",
        "Note: *freq* has been previously generated above for the programming task.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "collapsed": true,
        "id": "v6WAZaA8qiFL",
        "outputId": "23e4d2ad-ed41-4e2a-d428-d70156edcdfc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Frequency of lesion:   1\n",
            "Frequency of lesions:  2\n",
            "Frequency of LESION:   0\n",
            "Frequency of LESIONS:  3\n"
          ]
        }
      ],
      "source": [
        "print(\"Frequency of lesion:  \", freq[\"lesion\"])\n",
        "print(\"Frequency of lesions: \", freq[\"lesions\"])\n",
        "print(\"Frequency of LESION:  \", freq[\"LESION\"])\n",
        "print(\"Frequency of LESIONS: \", freq[\"LESIONS\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "--kpd-5BqiFL"
      },
      "source": [
        "### Further analysis of interest"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UcooeMHwqiFL"
      },
      "source": [
        "Why not analyse the two documents further? You could have a play with other sentences in the Medical Note document. Or you could try POS tagging for the Biopsy Report!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "176cw4ZNqiFL"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}