Python_CrashCourse_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNU+RMXDAQ0B+Uhc/fr+QsQ",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ieg-dhr/NLP-Course4Humanities_2024/blob/main/Python_CrashCourse_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Processing Datasets in Notebooks\n",
        "\n",
        "\n",
        "Created by Sarah Oberbichler [![ORCID](https://info.orcid.org/wp-content/uploads/2019/11/orcid_16x16.png)](https://orcid.org/0000-0002-1031-2759)\n"
      ],
      "metadata": {
        "id": "Z92h4kZjqoBk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Importing Data to the Notebook\n",
        "\n",
        "In order to access our course data, we clone the course GitHub repository to this notebook. Do do so, run the *git clone* cell below:"
      ],
      "metadata": {
        "id": "b0p9NHUqGPfu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/ieg-dhr/NLP-Course4Humanities_2024.git"
      ],
      "metadata": {
        "id": "g9WMeLNvG4SI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, when clicking on the file symbol on the left side of the Colab interface, you'll see the \"Files\" pane open up. Navigate through this pane to find and access the documents in the course file named \"NLP-Course4Humanities_2024\". This folder should contain all the relevant materials for our Natural Language Processing course. You can browse through the files, open documents, and use the content directly in your Colab notebook. Go to the folder *datasets*, open it and copy the path for Data_erxercise_1.xlsx. To do so, click on the 3 points that appear when you hover over the document. Click on the 3 points and choose *copy path/Pfad kopieren*\n",
        "\n"
      ],
      "metadata": {
        "id": "8ikbpo4fNQ6L"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UQpLyh05GMSL"
      },
      "outputs": [],
      "source": [
        "# @markdown ##### If you copied the path, find the place where you can add it. Run the code and investigate the dataset\n",
        "import pandas as pd\n",
        "\n",
        "# Replace 'your_file.xlsx' with the actual path to your Excel file.\n",
        "df = pd.read_excel('')\n",
        "\n",
        "# Display the first few rows of the DataFrame to verify it's loaded correctly.\n",
        "df.head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Investigate your data\n",
        "\n",
        "Now we can investigate the data. First let's check how many rows the excel file has:"
      ],
      "metadata": {
        "id": "TmYFTXPIKeQl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Length of the DataFrame\n",
        "\n",
        "print(f\"The DataFrame has {df.shape[0]} rows.\")"
      ],
      "metadata": {
        "id": "YOHIdX7IKNtQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Only work with one of the column\n",
        "df['paper_title']"
      ],
      "metadata": {
        "id": "dIE9fxvHYKZo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the number of different newspaper titles\n",
        "num_unique_titles = df['paper_title'].nunique()\n",
        "\n",
        "print(f\"There are {num_unique_titles} different paper titles in the dataset.\")"
      ],
      "metadata": {
        "id": "R4VRrXV-hkrp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# @markdown ##### **Exercise 1:** Now find the number of different places of distribution using the same method with the .nunique() function\n",
        "\n",
        "# add your code here"
      ],
      "metadata": {
        "id": "pHOMxQ_wXUNY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# @markdown ##### **Exercise 2:** Find the column with the full text, load the first three rows of this column. Do you remember from the first Crash Course how to find the first three letters in a string. We can use the same method to find the first three rows in a dataframe.\n",
        "\n",
        "# add your code here\n"
      ],
      "metadata": {
        "id": "WfND8IrQYU43"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# print full text of one row of plainpagefulltext\n",
        "\n",
        "print(df['plainpagefulltext'][0])"
      ],
      "metadata": {
        "id": "_Lk7NmzSaiU6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Change the size of your DataFrame\n",
        "\n",
        "We can also change the size of a DataFrame. Let's say we want to do some testing with only the first 10 rows of the dataset."
      ],
      "metadata": {
        "id": "9FD4B1_scKO3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Reduce your DataFrame to 10 rows\n",
        "df_short = df[:10]"
      ],
      "metadata": {
        "id": "0QmuubKfcx_L"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# @markdown ##### **Exercise 2:** Check if the DataFrame had been reduced to 10 rows\n",
        "\n",
        "#add your code here"
      ],
      "metadata": {
        "id": "zNNyjV43drXe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Exercise 4:\n",
        "\n",
        "Create two different DataFrames form the dataframe we were working with so far. Call them df_1 and df_2. Df_1 contains the first 100 rows, df_2 rows 101-200. After doing that, merge the two DataFrames again by using the *df.merge(df)* function."
      ],
      "metadata": {
        "id": "ExfWD8Zhetwz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Write your code here:\n",
        "\n"
      ],
      "metadata": {
        "id": "XX-PR1d_fbZR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Search through your Dataframe\n",
        "\n",
        "We might also want to only works a specific aspect from our dataset. Let's assume, we only want to use newspapers that have been distributed in Berlin."
      ],
      "metadata": {
        "id": "1BM7sPXUjU6M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Filter the DataFrame to include only rows where 'place_of_distribution' is 'Berlin'.\n",
        "df_berlin = df[df['place_of_distribution'].str.contains('Leipzig', na = False)]\n",
        "\n",
        "# Display the filtered DataFrame.\n",
        "df_berlin.head()\n",
        "\n"
      ],
      "metadata": {
        "id": "7pQ2WLETjr7W"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# How many rows are these?\n",
        "\n",
        "#add code here"
      ],
      "metadata": {
        "id": "mQqeXQGOk8hA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercise 5:\n",
        "\n",
        "Create a new DataFrame where each fulltext contains a specific word. For example \"Katastrophe\"."
      ],
      "metadata": {
        "id": "zDDBXhnLmqu7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#add your code here:\n",
        "\n"
      ],
      "metadata": {
        "id": "JaVkjnZOqS3Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercise 6:\n",
        "Use the generate function for this exercise --> Use the df_berlin dataset and search the full text. How often appears \"Berlin\" in the full text? What functions do you recognize from the Crash Course 1? Why do we have to start with zero?"
      ],
      "metadata": {
        "id": "MVIgVsIglOtt"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "SMGCz2gtnX1W"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}