select_CBC.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract only CBC news and only news\n",
    "For paper with Monika Bednarek. From each month of GGT output, extract only CBC articles. \n",
    "\n",
    "Also remove articles that GGT categorized as opinion (from parsing the URL). This still has a lot of opinion, so there is another notebook which filters first person, analysis, and opinion. I should probably merge them, but maybe later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "from pandas import json_normalize\n",
    "import ast\n",
    "from ast import literal_eval\n",
    "import os\n",
    "import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(r'C:\\Maite\\MOD\\papers\\Discourse_Communication\\data\\GGT_2023_by_month')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One month at a time\n",
    "\n",
    "There's probably a more efficient way of doing this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('2023_11_nov.csv', encoding = 'utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract the right information\n",
    "\n",
    "First the rows that are only:\n",
    "* CBC\n",
    "* hard news vs. opinion, as extracted by the GGT\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_CBC = df.loc[df[\"outlet\"] == 'CBC News']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_CBC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_CBC_hard = df_CBC.loc[df_CBC[\"articleType\"] == 'hard news']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_CBC_hard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save only the columns I want\n",
    "\n",
    "df_selection = df_CBC_hard[['_id', 'url', 'body', 'title']].copy().dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Remember to change file names here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save new csv\n",
    "\n",
    "os.chdir(r'C:\\Maite\\MOD\\projects\\Monika_Bednarek\\Evaluation_quotes\\Data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CHANGE THE NAME OF THE MONTH EACH TIME!\n",
    "\n",
    "df_selection.to_csv('2023_11_nov_CBC_hardnews.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concatenate all csvs - once I have produced all 12 months\n",
    "\n",
    "Once I have all the months, concatenate them and count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "extension = 'csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_filenames = [i for i in glob.glob('*.{}'.format(extension))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#combine all files in the list\n",
    "\n",
    "CBC_hardnews_all_2023 = pd.concat([pd.read_csv(f) for f in all_filenames])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CBC_hardnews_all_2023"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export to csv\n",
    "\n",
    "CBC_hardnews_all_2023.to_csv( \"CBC_hardnews_all_2023.csv\", index=False, encoding='utf-8-sig')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A sanity check, just check how many are opinion\n",
    "\n",
    "Note: this turns out not to be the only place where opinion is marked. Another notebook uses the URL structure to filter more opinion out. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(r'C:\\Maite\\MOD\\papers\\Discourse_Communication\\data\\GGT_2023_by_month')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = os.getcwd() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_files = glob.glob(os.path.join(path, \"*.csv\")) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining an empty list to store content\n",
    "df_all = pd.DataFrame() \n",
    "content_all = [] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read all the files into content, appending\n",
    "for f in csv_files: \n",
    "    df_temp = pd.read_csv(f, index_col=None) \n",
    "    content_all.append(df_temp) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now content into dataframe\n",
    "df_all = pd.concat(content_all) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_CBC_all = df_all.loc[df_all[\"outlet\"] == 'CBC News']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_CBC_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_CBC_opinion_all = df_CBC_all.loc[df_CBC_all[\"articleType\"] == 'opinion']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_CBC_opinion_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save to other project dir\n",
    "\n",
    "os.chdir(r'C:\\Maite\\MOD\\projects\\Monika_Bednarek\\Evaluation_quotes\\Data\\CBC_input')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export to csv\n",
    "\n",
    "df_CBC_opinion_all.to_csv( \"CBC_opinion_all_2023.csv\", index=False, encoding='utf-8-sig')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}