05 Data Cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python training for data engineers\n",
    "## 05. Data cleaning\n",
    "\n",
    "### Goal\n",
    "* Read the XML data from the previous notebook and add additional columns\n",
    "* Read the JSON data and retrieve related information\n",
    "\n",
    "### Remember\n",
    "* `xmltree = lxml.html.fromstring(content)` will create the XML tree from an XML string\n",
    "* `xmltree.cssselect('div[class=\\'important\\'] a')` will retrieve all links within the div with class 'important'\n",
    "* `df['col_name'].astype('category')` converts a dataframe column to 'category'\n",
    "* `df.fillna(0)` will replace empty values with the integer 0\n",
    "* `df['text'].str.extract(\"(\\d*)\", expand=True)` will extract a number from the text column\n",
    "* `df['new_column'] = df['text'].apply(lambda row: row[0], axis=1)` will retrieve the first character of the text column and add it to the new column\n",
    "\n",
    "### XML\n",
    "In the first example we will deal with the XML response of the first request. Import lxml to handle the XML response we got in the web crawling step.\n",
    "\n",
    "Make sure `lxml` is installed, or run `conda install lxml` first to install the library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load data from previous step\n",
    "import pickle\n",
    "xmlcontent = pickle.load(open(\"xmlcontent_notebook_04.pickle\", \"rb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lxml.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and construct the [XML tree](https://www.w3schools.com/xml/xml_tree.asp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Element html at 0x1122209a8>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmltree = lxml.html.fromstring(xmlcontent)\n",
    "xmltree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HTML source for the result page is shown in the following snippet.\n",
    "```html\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "<head>\n",
    "  <title></title>\n",
    "</head>\n",
    "<body>\n",
    "  ...  \n",
    "  <div class=\"package-snippet\">\n",
    "    <h3 class=\"package-snippet__title\">\n",
    "      <a href=\"/project/learning/\">Learning</a>\n",
    "      <span class=\"package-snippet__version\">1.0.0</span>\n",
    "    </h3>\n",
    "    <p class=\"package-snippet__description\">Tyro</p>\n",
    "  </div>\n",
    "    ...\n",
    "</body>\n",
    "</html>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `tag` command we can see the top element of the XML tree, which indeed is the first element as we expect from the XML above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'html'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmltree.tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to get all links from the table and add them to a list.\n",
    "\n",
    "```html\n",
    "<div class=\"package-snippet\"><h3><a href=\"link\"></a></h3>\n",
    "```\n",
    "\n",
    "Make sure `cssselect` is installed inside the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/project/scikit-learn/\n",
      "/project/scikit-build/\n",
      "/project/scikit-ci-addons/\n",
      "/project/scikit-hep/\n",
      "/project/scikit-cycling/\n",
      "/project/scikit-ci/\n",
      "/project/scikit-learn-runnr/\n",
      "/project/scikit-allel/\n",
      "/project/scikit-chem/\n",
      "/project/scikit-vis/\n",
      "/project/scikit-bio/\n",
      "/project/scikit-monaco/\n",
      "/project/scikits-learn/\n",
      "/project/scikit-optimize/\n",
      "/project/scikit-ribo/\n",
      "/project/scikit-dataaccess/\n",
      "/project/ninja/\n",
      "/project/scikit-ued/\n",
      "/project/scikit-nano/\n",
      "/project/scikit-neuralnetwork/\n"
     ]
    }
   ],
   "source": [
    "for link in xmltree.cssselect('div[class=\\'package-snippet\\'] h3 a'):\n",
    "    print(link.get('href'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets save them to an array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an empty list\n",
    "list_of_links = []\n",
    "# Loop through all the links in the table\n",
    "for link in xmltree.cssselect('div[class=\\'package-snippet\\'] h3 a'):\n",
    "    # Prepend with full URL and remove the version tag at the end by splitting by '/' and taking the relevant parts\n",
    "    list_of_links.append('https://pypi.python.org' + link.get('href').rsplit('/',1)[0]+'/#files')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the first element of the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://pypi.python.org/project/scikit-learn/#files'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_of_links[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting detailed information per package\n",
    "Next we will loop over the links we retrieved in the previous step. Per detail page we will extract information about the packages.\n",
    "\n",
    "Example URL: https://pypi.python.org/pypi/scikit-learn/0.19.0/#files\n",
    "\n",
    "```html\n",
    "<table class=\"table table--downloads\">\n",
    "    <thead>\n",
    "        <tr>\n",
    "            <th class=\"table__filename\">\n",
    "                    Filename, size &amp; hash\n",
    "                    <a href=\"https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode\" class=\"tooltipped tooltipped-n\" aria-label=\"what's this?\" data-original-label=\"what's this?\" target=\"_blank\"><i class=\"fa fa-question-circle\" aria-hidden=\"true\"></i><span class=\"sr-only\">SHA256 hash help</span></a></th>\n",
    "            <th class=\"table__type\">File type</th>\n",
    "            <th class=\"table__version\">Python version</th>\n",
    "            <th class=\"table__upload-date\">Upload date</th>\n",
    "        </tr>\n",
    "    </thead>\n",
    "    <tbody>\n",
    "        <tr>\n",
    "            <td>\n",
    "                <a href=\"https://files.pythonhosted.org/packages/de/d3/47c2c9842d61042f3c5f082f677dbe05899b077272105906a3249fe8c5da/scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl\">\n",
    "                      scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl\n",
    "                    </a>\n",
    "                     (8.0 MB)\n",
    "                    <a class=\"-js-copy-hash table__sha256-link tooltipped tooltipped-s\" aria-label=\"Copy to clipboard\" data-original-label=\"Copy to clipboard\" data-clipboard-text=\"3775cca4ce3f94508bb7c8a6b113044b78c16b0a30a5c169ddeb6b9fe57a8a72\"><i class=\"fa fa-copy\" aria-hidden=\"true\"></i><span class=\"sr-only\">Copy SHA256 hash</span>\n",
    "                      SHA256\n",
    "                    </a></td>\n",
    "            <td>\n",
    "                    Wheel\n",
    "                  </td>\n",
    "            <td>\n",
    "                    \n",
    "                      cp27\n",
    "                    \n",
    "                  </td>\n",
    "            <td>Oct 23, 2017</td>\n",
    "        </tr>\n",
    "    </tbody>\n",
    "</table>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieve information for the first link by requesting the URL and convert it to an XML tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "response = requests.get(list_of_links[0])\n",
    "xmltree = lxml.html.fromstring(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will go through the table as depicted in the HTML snippet above, and extract the relevant information.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'filename_size_hash': '\\n                    \\n                      scikit-learn-0.19.1.win-amd64-py2.7.exe\\n                    \\n                     (4.7 MB)\\n                    \\n                      \\n                      Copy SHA256 hash\\n                      SHA256\\n                    \\n                  ',\n",
       " 'file_type': '\\n                    Windows Installer\\n                  ',\n",
       " 'python_version': '\\n                    \\n                      2.7\\n                    \\n                  ',\n",
       " 'uploaded_on': 'Oct 23, 2017'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create dictionary\n",
    "module_info = {}\n",
    "for entry in xmltree.cssselect('table[class*=\\'table--downloads\\'] tr')[1:-1]:\n",
    "    module_info['filename_size_hash'] = entry.cssselect('td')[0].text_content()\n",
    "    module_info['file_type'] = entry.cssselect('td')[1].text_content()\n",
    "    module_info['python_version'] = entry.cssselect('td')[2].text_content()\n",
    "    module_info['uploaded_on'] = entry.cssselect('td')[3].text_content()\n",
    "module_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see in the `module_info` different values have weird characters. We need to remove spaces and new line characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'filename_size_hash': 'scikit-learn-0.19.1.win-amd64-py2.7.exe                                         (4.7 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       " 'file_type': 'windows installer',\n",
       " 'python_version': '2.7',\n",
       " 'uploaded_on': 'Oct 23, 2017'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module_info = {}\n",
    "for entry in xmltree.cssselect('table[class*=\\'table--downloads\\'] tr')[1:-1]:\n",
    "    module_info['filename_size_hash'] = entry.cssselect('td')[0].text_content().strip().replace('\\n', '')\n",
    "    module_info['file_type'] = entry.cssselect('td')[1].text_content().strip().replace('\\n', '').lower()\n",
    "    module_info['python_version'] = entry.cssselect('td')[2].text_content().strip().replace('\\n', '')\n",
    "    module_info['uploaded_on'] = entry.cssselect('td')[3].text_content()\n",
    "module_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets put this in a function, so we can simply extract the data by calling the function with the URL's for the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_info_from_link(url):\n",
    "    response = requests.get(url)\n",
    "    xmltree = lxml.html.fromstring(response.content)\n",
    "    module_info_list = []\n",
    "    for entry in xmltree.cssselect('table[class*=\\'table--downloads\\'] tr')[1:-1]:\n",
    "        module_info = {}\n",
    "        module_info['package_name'] = url.rsplit('/',2)[1]\n",
    "        module_info['filename_size_hash'] = entry.cssselect('td')[0].text_content().strip().replace('\\n', '')\n",
    "        module_info['file_type'] = entry.cssselect('td')[1].text_content().strip().replace('\\n', '').lower()\n",
    "        try:\n",
    "            module_info['python_version'] = entry.cssselect('td')[2].text_content().strip().replace('\\n', '')\n",
    "        except:\n",
    "            pass # No version found\n",
    "        try:\n",
    "            module_info['uploaded_on'] = entry.cssselect('td')[3].text_content()\n",
    "        except:\n",
    "            pass # No date found\n",
    "        module_info_list.append(module_info)\n",
    "    return module_info_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the function for the first link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (8.0 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686.whl                                         (11.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_64.whl                                         (12.2 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i686.whl                                         (11.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl                                         (12.2 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-win32.whl                                         (4.1 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-win_amd64.whl                                         (4.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (7.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-manylinux1_i686.whl                                         (11.6 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-manylinux1_x86_64.whl                                         (12.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-win32.whl                                         (4.0 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-win_amd64.whl                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (7.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-manylinux1_i686.whl                                         (11.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-manylinux1_x86_64.whl                                         (12.2 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-win32.whl                                         (3.9 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-win_amd64.whl                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (7.6 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-manylinux1_i686.whl                                         (11.6 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-manylinux1_x86_64.whl                                         (12.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-win32.whl                                         (3.9 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-win_amd64.whl                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit-learn-0.19.1.tar.gz                                         (9.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'source',\n",
       "  'python_version': 'None',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit-learn-0.19.1.win32-py2.7.exe                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'windows installer',\n",
       "  'python_version': '2.7',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit-learn-0.19.1.win32-py3.5.exe                                         (4.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'windows installer',\n",
       "  'python_version': '3.5',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit-learn-0.19.1.win-amd64-py2.7.exe                                         (4.7 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'windows installer',\n",
       "  'python_version': '2.7',\n",
       "  'uploaded_on': 'Oct 23, 2017'}]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "info = get_info_from_link(list_of_links[0])\n",
    "info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the info shown above, we can conclude that there are different versions for the module available.\n",
    "\n",
    "Lets extract data for the second link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'package_name': 'scikit-build',\n",
       "  'filename_size_hash': 'scikit_build-0.6.1-py2.py3-none-any.whl                                         (52.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'py2.py3',\n",
       "  'uploaded_on': 'Jun 8, 2017'}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "info_two = get_info_from_link(list_of_links[1])\n",
    "info_two"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now combine the two information blocks into one list by creating a new list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (8.0 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp27',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686.whl                                         (11.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp27',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_64.whl                                         (12.2 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp27',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i686.whl                                         (11.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp27',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl                                         (12.2 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp27',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-win32.whl                                         (4.1 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp27',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-win_amd64.whl                                         (4.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp27',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (7.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp34',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-manylinux1_i686.whl                                         (11.6 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp34',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-manylinux1_x86_64.whl                                         (12.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp34',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-win32.whl                                         (4.0 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp34',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-win_amd64.whl                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp34',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (7.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp35',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-manylinux1_i686.whl                                         (11.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp35',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-manylinux1_x86_64.whl                                         (12.2 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp35',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-win32.whl                                         (3.9 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp35',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-win_amd64.whl                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp35',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (7.6 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp36',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-manylinux1_i686.whl                                         (11.6 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp36',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-manylinux1_x86_64.whl                                         (12.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp36',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-win32.whl                                         (3.9 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp36',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-win_amd64.whl                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'cp36',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit-learn-0.19.1.tar.gz                                         (9.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'source',\n",
       "   'python_version': 'None',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit-learn-0.19.1.win32-py2.7.exe                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'windows installer',\n",
       "   'python_version': '2.7',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit-learn-0.19.1.win32-py3.5.exe                                         (4.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'windows installer',\n",
       "   'python_version': '3.5',\n",
       "   'uploaded_on': 'Oct 23, 2017'},\n",
       "  {'package_name': 'scikit-learn',\n",
       "   'filename_size_hash': 'scikit-learn-0.19.1.win-amd64-py2.7.exe                                         (4.7 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'windows installer',\n",
       "   'python_version': '2.7',\n",
       "   'uploaded_on': 'Oct 23, 2017'}],\n",
       " [{'package_name': 'scikit-build',\n",
       "   'filename_size_hash': 'scikit_build-0.6.1-py2.py3-none-any.whl                                         (52.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "   'file_type': 'wheel',\n",
       "   'python_version': 'py2.py3',\n",
       "   'uploaded_on': 'Jun 8, 2017'}]]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_info = [info, info_two]\n",
    "total_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets extract the information for <u>all</u> the links to make a more interesting dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (8.0 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686.whl                                         (11.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_64.whl                                         (12.2 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i686.whl                                         (11.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl                                         (12.2 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-win32.whl                                         (4.1 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp27-cp27m-win_amd64.whl                                         (4.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (7.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-manylinux1_i686.whl                                         (11.6 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-manylinux1_x86_64.whl                                         (12.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-win32.whl                                         (4.0 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp34-cp34m-win_amd64.whl                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (7.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-manylinux1_i686.whl                                         (11.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-manylinux1_x86_64.whl                                         (12.2 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-win32.whl                                         (3.9 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp35-cp35m-win_amd64.whl                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (7.6 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-manylinux1_i686.whl                                         (11.6 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-manylinux1_x86_64.whl                                         (12.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-win32.whl                                         (3.9 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit_learn-0.19.1-cp36-cp36m-win_amd64.whl                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit-learn-0.19.1.tar.gz                                         (9.5 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'source',\n",
       "  'python_version': 'None',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit-learn-0.19.1.win32-py2.7.exe                                         (4.3 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'windows installer',\n",
       "  'python_version': '2.7',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit-learn-0.19.1.win32-py3.5.exe                                         (4.4 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'windows installer',\n",
       "  'python_version': '3.5',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-learn',\n",
       "  'filename_size_hash': 'scikit-learn-0.19.1.win-amd64-py2.7.exe                                         (4.7 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'windows installer',\n",
       "  'python_version': '2.7',\n",
       "  'uploaded_on': 'Oct 23, 2017'},\n",
       " {'package_name': 'scikit-build',\n",
       "  'filename_size_hash': 'scikit_build-0.6.1-py2.py3-none-any.whl                                         (52.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'py2.py3',\n",
       "  'uploaded_on': 'Jun 8, 2017'},\n",
       " {'package_name': 'scikit-ci-addons',\n",
       "  'filename_size_hash': 'scikit_ci_addons-0.20.0-py2.py3-none-any.whl                                         (33.2 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'py2.py3',\n",
       "  'uploaded_on': 'Nov 16, 2017'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (1.9 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp27-cp27m-manylinux1_i686.whl                                         (927.0 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp27-cp27m-manylinux1_x86_64.whl                                         (939.7 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp27-cp27mu-manylinux1_i686.whl                                         (926.8 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp27-cp27mu-manylinux1_x86_64.whl                                         (937.7 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp27-cp27m-win32.whl                                         (882.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp27-cp27m-win_amd64.whl                                         (899.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (2.0 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp35-cp35m-manylinux1_i686.whl                                         (927.0 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp35-cp35m-manylinux1_x86_64.whl                                         (940.0 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp35-cp35m-win32.whl                                         (881.4 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp35-cp35m-win_amd64.whl                                         (899.0 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl                                         (2.0 MB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp36-cp36m-manylinux1_i686.whl                                         (928.0 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp36-cp36m-manylinux1_x86_64.whl                                         (940.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp36-cp36m-win32.whl                                         (882.7 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-cycling',\n",
       "  'filename_size_hash': 'scikit_cycling-0.1.3-cp36-cp36m-win_amd64.whl                                         (899.8 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Mar 19, 2018'},\n",
       " {'package_name': 'scikit-ci',\n",
       "  'filename_size_hash': 'scikit_ci-0.15.0-py2.py3-none-any.whl                                         (13.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'py2.py3',\n",
       "  'uploaded_on': 'Jun 30, 2017'},\n",
       " {'package_name': 'scikit-chem',\n",
       "  'filename_size_hash': 'scikit_chem-0.0.6-py2.py3-none-any.whl                                         (102.0 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': '2.7',\n",
       "  'uploaded_on': 'Aug 8, 2016'},\n",
       " {'package_name': 'scikit-chem',\n",
       "  'filename_size_hash': 'scikit_chem-0.0.6-py3-none-any.whl                                         (102.0 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': '3.5',\n",
       "  'uploaded_on': 'Aug 8, 2016'},\n",
       " {'package_name': 'scikit-optimize',\n",
       "  'filename_size_hash': 'scikit_optimize-0.5.2-py2.py3-none-any.whl                                         (74.5 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'py2.py3',\n",
       "  'uploaded_on': 'Mar 25, 2018'},\n",
       " {'package_name': 'scikit-ribo',\n",
       "  'filename_size_hash': 'scikit_ribo-0.2.4b1-py3-none-any.whl                                         (73.2 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'py3',\n",
       "  'uploaded_on': 'Jun 27, 2017'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp27-cp27m-macosx_10_6_x86_64.whl                                         (83.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp27-cp27m-manylinux1_i686.whl                                         (88.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp27-cp27m-manylinux1_x86_64.whl                                         (88.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp27-cp27mu-manylinux1_i686.whl                                         (88.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp27-cp27mu-manylinux1_x86_64.whl                                         (88.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp27-cp27m-win32.whl                                         (173.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp27-cp27m-win_amd64.whl                                         (253.3 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp27',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp33-cp33m-macosx_10_6_x86_64.whl                                         (85.7 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp33',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp33-cp33m-manylinux1_i686.whl                                         (91.0 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp33',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp33-cp33m-manylinux1_x86_64.whl                                         (90.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp33',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp33-cp33m-win32.whl                                         (165.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp33',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp33-cp33m-win_amd64.whl                                         (255.3 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp33',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp34-cp34m-macosx_10_6_x86_64.whl                                         (83.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp34-cp34m-manylinux1_i686.whl                                         (88.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp34-cp34m-manylinux1_x86_64.whl                                         (88.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp34-cp34m-win32.whl                                         (163.5 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp34-cp34m-win_amd64.whl                                         (253.2 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp34',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp35-cp35m-macosx_10_6_x86_64.whl                                         (83.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp35-cp35m-manylinux1_i686.whl                                         (88.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp35-cp35m-manylinux1_x86_64.whl                                         (88.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp35-cp35m-win32.whl                                         (210.4 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp35-cp35m-win_amd64.whl                                         (253.2 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp35',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp36-cp36m-manylinux1_i686.whl                                         (88.9 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp36-cp36m-manylinux1_x86_64.whl                                         (88.6 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp36-cp36m-win32.whl                                         (210.4 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Apr 8, 2018'},\n",
       " {'package_name': 'ninja',\n",
       "  'filename_size_hash': 'ninja-1.8.2-cp36-cp36m-win_amd64.whl                                         (253.2 kB)                                                                Copy SHA256 hash                      SHA256',\n",
       "  'file_type': 'wheel',\n",
       "  'python_version': 'cp36',\n",
       "  'uploaded_on': 'Apr 8, 2018'}]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize an empty list\n",
    "all_info = []\n",
    "for link in list_of_links:\n",
    "    # Extract the data\n",
    "    info_list = get_info_from_link(link)\n",
    "    # Append the info to the big list\n",
    "    all_info += info_list\n",
    "all_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data conversion\n",
    "Import pandas so we can start using dataframes with our retrieved data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_type</th>\n",
       "      <th>filename_size_hash</th>\n",
       "      <th>package_name</th>\n",
       "      <th>python_version</th>\n",
       "      <th>uploaded_on</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>Oct 23, 2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>Oct 23, 2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>Oct 23, 2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>Oct 23, 2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>Oct 23, 2017</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  file_type                                 filename_size_hash  package_name  \\\n",
       "0     wheel  scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...  scikit-learn   \n",
       "1     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...  scikit-learn   \n",
       "2     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...  scikit-learn   \n",
       "3     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...  scikit-learn   \n",
       "4     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...  scikit-learn   \n",
       "\n",
       "  python_version   uploaded_on  \n",
       "0           cp27  Oct 23, 2017  \n",
       "1           cp27  Oct 23, 2017  \n",
       "2           cp27  Oct 23, 2017  \n",
       "3           cp27  Oct 23, 2017  \n",
       "4           cp27  Oct 23, 2017  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmldf = pd.DataFrame.from_dict(all_info)\n",
    "xmldf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the datatypes of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "file_type             object\n",
       "filename_size_hash    object\n",
       "package_name          object\n",
       "python_version        object\n",
       "uploaded_on           object\n",
       "dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmldf.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the `release_type` to a category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "xmldf['file_type'] = xmldf['file_type'].astype('category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the `uploaded_on` to a proper timestamp using `pd.to_datetime()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "xmldf['uploaded_on'] = pd.to_datetime(xmldf['uploaded_on'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the `filename_size_hash` into filename and size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "xmldf[['filename', 'size']] = xmldf['filename_size_hash'].str.extract('(\\w.*?)\\((.*?)\\)', expand=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_type</th>\n",
       "      <th>filename_size_hash</th>\n",
       "      <th>package_name</th>\n",
       "      <th>python_version</th>\n",
       "      <th>uploaded_on</th>\n",
       "      <th>filename</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...</td>\n",
       "      <td>8.0 MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...</td>\n",
       "      <td>11.4 MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...</td>\n",
       "      <td>12.2 MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...</td>\n",
       "      <td>11.4 MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...</td>\n",
       "      <td>12.2 MB</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  file_type                                 filename_size_hash  package_name  \\\n",
       "0     wheel  scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...  scikit-learn   \n",
       "1     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...  scikit-learn   \n",
       "2     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...  scikit-learn   \n",
       "3     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...  scikit-learn   \n",
       "4     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...  scikit-learn   \n",
       "\n",
       "  python_version uploaded_on  \\\n",
       "0           cp27  2017-10-23   \n",
       "1           cp27  2017-10-23   \n",
       "2           cp27  2017-10-23   \n",
       "3           cp27  2017-10-23   \n",
       "4           cp27  2017-10-23   \n",
       "\n",
       "                                            filename     size  \n",
       "0  scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...   8.0 MB  \n",
       "1  scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...  11.4 MB  \n",
       "2  scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...  12.2 MB  \n",
       "3  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...  11.4 MB  \n",
       "4  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...  12.2 MB  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmldf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the `size` into a `size` and a `unit` by using a regular expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "xmldf[['size', 'unit']] = xmldf['size'].str.extract('(\\d*)\\s(\\w*?)$', expand=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_type</th>\n",
       "      <th>filename_size_hash</th>\n",
       "      <th>package_name</th>\n",
       "      <th>python_version</th>\n",
       "      <th>uploaded_on</th>\n",
       "      <th>filename</th>\n",
       "      <th>size</th>\n",
       "      <th>unit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...</td>\n",
       "      <td>0</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...</td>\n",
       "      <td>4</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...</td>\n",
       "      <td>2</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...</td>\n",
       "      <td>4</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...</td>\n",
       "      <td>2</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  file_type                                 filename_size_hash  package_name  \\\n",
       "0     wheel  scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...  scikit-learn   \n",
       "1     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...  scikit-learn   \n",
       "2     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...  scikit-learn   \n",
       "3     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...  scikit-learn   \n",
       "4     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...  scikit-learn   \n",
       "\n",
       "  python_version uploaded_on  \\\n",
       "0           cp27  2017-10-23   \n",
       "1           cp27  2017-10-23   \n",
       "2           cp27  2017-10-23   \n",
       "3           cp27  2017-10-23   \n",
       "4           cp27  2017-10-23   \n",
       "\n",
       "                                            filename size unit  \n",
       "0  scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...    0   MB  \n",
       "1  scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...    4   MB  \n",
       "2  scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...    2   MB  \n",
       "3  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...    4   MB  \n",
       "4  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...    2   MB  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmldf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fill the empty `size` values with a 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "xmldf['size'] = xmldf['size'].fillna(0).astype('int')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the datatypes again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "file_type                   category\n",
       "filename_size_hash            object\n",
       "package_name                  object\n",
       "python_version                object\n",
       "uploaded_on           datetime64[ns]\n",
       "filename                      object\n",
       "size                           int64\n",
       "unit                          object\n",
       "dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmldf.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_type</th>\n",
       "      <th>filename_size_hash</th>\n",
       "      <th>package_name</th>\n",
       "      <th>python_version</th>\n",
       "      <th>uploaded_on</th>\n",
       "      <th>filename</th>\n",
       "      <th>size</th>\n",
       "      <th>unit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...</td>\n",
       "      <td>0</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...</td>\n",
       "      <td>4</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...</td>\n",
       "      <td>2</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...</td>\n",
       "      <td>4</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...</td>\n",
       "      <td>2</td>\n",
       "      <td>MB</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  file_type                                 filename_size_hash  package_name  \\\n",
       "0     wheel  scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...  scikit-learn   \n",
       "1     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...  scikit-learn   \n",
       "2     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...  scikit-learn   \n",
       "3     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...  scikit-learn   \n",
       "4     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...  scikit-learn   \n",
       "\n",
       "  python_version uploaded_on  \\\n",
       "0           cp27  2017-10-23   \n",
       "1           cp27  2017-10-23   \n",
       "2           cp27  2017-10-23   \n",
       "3           cp27  2017-10-23   \n",
       "4           cp27  2017-10-23   \n",
       "\n",
       "                                            filename  size unit  \n",
       "0  scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...     0   MB  \n",
       "1  scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...     4   MB  \n",
       "2  scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...     2   MB  \n",
       "3  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...     4   MB  \n",
       "4  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...     2   MB  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmldf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lambda function\n",
    "Define a function to convert the size to bytes based on the unit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_to_bytes(size, unit):\n",
    "    if unit == 'kB':\n",
    "        size = size*1024\n",
    "    elif unit == 'MB':\n",
    "        size = size*1024*1024\n",
    "    else:\n",
    "        size = size\n",
    "    return size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply the function to convert the sizes using the `lambda` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file_type</th>\n",
       "      <th>filename_size_hash</th>\n",
       "      <th>package_name</th>\n",
       "      <th>python_version</th>\n",
       "      <th>uploaded_on</th>\n",
       "      <th>filename</th>\n",
       "      <th>size</th>\n",
       "      <th>unit</th>\n",
       "      <th>size_in_bytes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...</td>\n",
       "      <td>0</td>\n",
       "      <td>MB</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...</td>\n",
       "      <td>4</td>\n",
       "      <td>MB</td>\n",
       "      <td>4194304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...</td>\n",
       "      <td>2</td>\n",
       "      <td>MB</td>\n",
       "      <td>2097152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...</td>\n",
       "      <td>4</td>\n",
       "      <td>MB</td>\n",
       "      <td>4194304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>wheel</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...</td>\n",
       "      <td>scikit-learn</td>\n",
       "      <td>cp27</td>\n",
       "      <td>2017-10-23</td>\n",
       "      <td>scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...</td>\n",
       "      <td>2</td>\n",
       "      <td>MB</td>\n",
       "      <td>2097152</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  file_type                                 filename_size_hash  package_name  \\\n",
       "0     wheel  scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...  scikit-learn   \n",
       "1     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...  scikit-learn   \n",
       "2     wheel  scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...  scikit-learn   \n",
       "3     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...  scikit-learn   \n",
       "4     wheel  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...  scikit-learn   \n",
       "\n",
       "  python_version uploaded_on  \\\n",
       "0           cp27  2017-10-23   \n",
       "1           cp27  2017-10-23   \n",
       "2           cp27  2017-10-23   \n",
       "3           cp27  2017-10-23   \n",
       "4           cp27  2017-10-23   \n",
       "\n",
       "                                            filename  size unit  size_in_bytes  \n",
       "0  scikit_learn-0.19.1-cp27-cp27m-macosx_10_6_int...     0   MB              0  \n",
       "1  scikit_learn-0.19.1-cp27-cp27m-manylinux1_i686...     4   MB        4194304  \n",
       "2  scikit_learn-0.19.1-cp27-cp27m-manylinux1_x86_...     2   MB        2097152  \n",
       "3  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_i68...     4   MB        4194304  \n",
       "4  scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86...     2   MB        2097152  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmldf['size_in_bytes'] = xmldf.apply(lambda row: convert_to_bytes(row['size'], row['unit']), axis=1)\n",
    "xmldf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "file_type                   category\n",
       "filename_size_hash            object\n",
       "package_name                  object\n",
       "python_version                object\n",
       "uploaded_on           datetime64[ns]\n",
       "filename                      object\n",
       "size                           int64\n",
       "unit                          object\n",
       "size_in_bytes                  int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xmldf.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "xmldf.to_pickle('xml_dataframe_notebook_05.pickle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling the JSON data\n",
    "Lets read the JSON data into a dataframe via a pickle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "jsoncontent = pickle.load(open(\"jsoncontent_notebook_04.pickle\", \"rb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'items': [{'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 940527,\n",
       "   'name': 'python'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 86864,\n",
       "   'name': 'django'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 62823,\n",
       "   'name': 'python-3.x'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 59204,\n",
       "   'name': 'pandas'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 53225,\n",
       "   'name': 'python-2.7'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 42574,\n",
       "   'name': 'numpy'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 30503,\n",
       "   'name': 'list'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 25829,\n",
       "   'name': 'matplotlib'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 21991,\n",
       "   'name': 'dictionary'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 20116,\n",
       "   'name': 'regex'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 17262,\n",
       "   'name': 'flask'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 17071,\n",
       "   'name': 'tkinter'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 16199,\n",
       "   'name': 'string'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 15832,\n",
       "   'name': 'csv'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 15563,\n",
       "   'name': 'arrays'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 14821,\n",
       "   'name': 'dataframe'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 13432,\n",
       "   'name': 'json'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 11852,\n",
       "   'name': 'tensorflow'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 11405,\n",
       "   'name': 'google-app-engine'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 11214,\n",
       "   'name': 'html'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 10657,\n",
       "   'name': 'mysql'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 10588,\n",
       "   'name': 'selenium'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 10477,\n",
       "   'name': 'beautifulsoup'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 10106,\n",
       "   'name': 'linux'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 9971,\n",
       "   'name': 'scipy'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 9326,\n",
       "   'name': 'multithreading'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 9145,\n",
       "   'name': 'opencv'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 9065,\n",
       "   'name': 'sqlalchemy'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 8940,\n",
       "   'name': 'javascript'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 8657,\n",
       "   'name': 'windows'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 8101,\n",
       "   'name': 'scikit-learn'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 8031,\n",
       "   'name': 'django-models'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 7919,\n",
       "   'name': 'function'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 7860,\n",
       "   'name': 'pyqt'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 7715,\n",
       "   'name': 'machine-learning'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 7697,\n",
       "   'name': 'web-scraping'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 7689,\n",
       "   'name': 'class'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 7354,\n",
       "   'name': 'pygame'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 7321,\n",
       "   'name': 'loops'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 7204,\n",
       "   'name': 'xml'},\n",
       "  {'has_synonyms': False,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 7013,\n",
       "   'name': 'scrapy'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 6986,\n",
       "   'name': 'pip'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 6766,\n",
       "   'name': 'macos'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 6726,\n",
       "   'name': 'file'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 6668,\n",
       "   'name': 'c++'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 6612,\n",
       "   'name': 'datetime'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 6488,\n",
       "   'name': 'algorithm'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 6237,\n",
       "   'name': 'sockets'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 6201,\n",
       "   'name': 'subprocess'},\n",
       "  {'has_synonyms': True,\n",
       "   'is_moderator_only': False,\n",
       "   'is_required': False,\n",
       "   'count': 6001,\n",
       "   'name': 'python-requests'}],\n",
       " 'has_more': False,\n",
       " 'quota_max': 300,\n",
       " 'quota_remaining': 148}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jsoncontent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to convert the list of items to a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>has_synonyms</th>\n",
       "      <th>is_moderator_only</th>\n",
       "      <th>is_required</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>940527</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>86864</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>django</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>62823</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>python-3.x</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>59204</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>pandas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>53225</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>python-2.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    count  has_synonyms  is_moderator_only  is_required        name\n",
       "0  940527          True              False        False      python\n",
       "1   86864         False              False        False      django\n",
       "2   62823          True              False        False  python-3.x\n",
       "3   59204         False              False        False      pandas\n",
       "4   53225         False              False        False  python-2.7"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jsondf = pd.DataFrame.from_dict(jsoncontent['items'])\n",
    "jsondf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe that the datatypes of the dataframe are inherited, in contrast to the XML experiment from the first section of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count                 int64\n",
       "has_synonyms           bool\n",
       "is_moderator_only      bool\n",
       "is_required            bool\n",
       "name                 object\n",
       "dtype: object"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jsondf.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets extract all the Python related tags from the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA = {}\n",
    "# Iterate over all the items\n",
    "for item in jsoncontent['items']:\n",
    "    # Check if python is absent in the DATA dictionary\n",
    "    if 'python' not in DATA.keys():\n",
    "        # Create a new empty dictionary for the python key in DATA\n",
    "        DATA['python'] = {}\n",
    "    # Add all non-python items to the python key\n",
    "    if item['name'] != 'python':\n",
    "        # Create a key in the python dictionary with the number of references\n",
    "        DATA['python'][item['name']] = item['count']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the first tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "django\n"
     ]
    }
   ],
   "source": [
    "for key in DATA['python'].keys():\n",
    "    print(key)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above key is one of the many items related to Python according to the StackOverflow API. Lets use this key to extract all tags related to the key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "API_URL = 'https://api.stackexchange.com/2.2/tags/%s/related?pagesize=100&site=stackoverflow' % key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieve the related tags for the first key found above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(API_URL)\n",
    "data = response.json()\n",
    "\n",
    "for item in data['items']:\n",
    "    if key not in DATA.keys():\n",
    "        DATA[key] = {}\n",
    "    if item['name'] != key:\n",
    "        DATA[key][item['name']] = item['count']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Current dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>django</th>\n",
       "      <th>python-3.x</th>\n",
       "      <th>pandas</th>\n",
       "      <th>python-2.7</th>\n",
       "      <th>numpy</th>\n",
       "      <th>list</th>\n",
       "      <th>matplotlib</th>\n",
       "      <th>dictionary</th>\n",
       "      <th>regex</th>\n",
       "      <th>flask</th>\n",
       "      <th>...</th>\n",
       "      <th>foreign-keys</th>\n",
       "      <th>django-authentication</th>\n",
       "      <th>django-haystack</th>\n",
       "      <th>django-cms</th>\n",
       "      <th>django-south</th>\n",
       "      <th>amazon-web-services</th>\n",
       "      <th>many-to-many</th>\n",
       "      <th>models</th>\n",
       "      <th>virtualenv</th>\n",
       "      <th>django-class-based-views</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>django</th>\n",
       "      <td>NaN</td>\n",
       "      <td>3657</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3659</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>1052.0</td>\n",
       "      <td>1048.0</td>\n",
       "      <td>1036.0</td>\n",
       "      <td>990.0</td>\n",
       "      <td>984.0</td>\n",
       "      <td>939.0</td>\n",
       "      <td>919.0</td>\n",
       "      <td>887.0</td>\n",
       "      <td>870.0</td>\n",
       "      <td>870.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>python</th>\n",
       "      <td>86864.0</td>\n",
       "      <td>62823</td>\n",
       "      <td>59204.0</td>\n",
       "      <td>53225</td>\n",
       "      <td>42574.0</td>\n",
       "      <td>30503.0</td>\n",
       "      <td>25829.0</td>\n",
       "      <td>21991.0</td>\n",
       "      <td>20116.0</td>\n",
       "      <td>17262.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 90 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         django  python-3.x   pandas  python-2.7    numpy     list  \\\n",
       "django      NaN        3657      NaN        3659      NaN      NaN   \n",
       "python  86864.0       62823  59204.0       53225  42574.0  30503.0   \n",
       "\n",
       "        matplotlib  dictionary    regex    flask            ...             \\\n",
       "django         NaN         NaN      NaN      NaN            ...              \n",
       "python     25829.0     21991.0  20116.0  17262.0            ...              \n",
       "\n",
       "        foreign-keys  django-authentication  django-haystack  django-cms  \\\n",
       "django        1052.0                 1048.0           1036.0       990.0   \n",
       "python           NaN                    NaN              NaN         NaN   \n",
       "\n",
       "        django-south  amazon-web-services  many-to-many  models  virtualenv  \\\n",
       "django         984.0                939.0         919.0   887.0       870.0   \n",
       "python           NaN                  NaN           NaN     NaN         NaN   \n",
       "\n",
       "        django-class-based-views  \n",
       "django                     870.0  \n",
       "python                       NaN  \n",
       "\n",
       "[2 rows x 90 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame.from_dict(DATA, orient='index')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can observe we are creating a 2 dimensional matrix with a count between the horizontal and vertical item.\n",
    "\n",
    "Show the existing keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['python', 'django'])\n"
     ]
    }
   ],
   "source": [
    "print(DATA.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets create a function for the code we just ran."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_items_from_key(key):\n",
    "    API_URL = 'https://api.stackexchange.com/2.2/tags/%s/related?pagesize=100&site=stackoverflow' % key\n",
    "    response = requests.get(API_URL)\n",
    "    data = response.json()\n",
    "\n",
    "    for item in data['items']:\n",
    "        if key not in DATA.keys():\n",
    "            DATA[key] = {}\n",
    "        if item['name'] != key:\n",
    "            DATA[key][item['name']] = item['count']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now iterate over the keys for Python and retrieve all the items per key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in DATA['python'].keys():\n",
    "    if (key not in DATA.keys()):\n",
    "        add_items_from_key(key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the final dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>django</th>\n",
       "      <th>python-3.x</th>\n",
       "      <th>pandas</th>\n",
       "      <th>python-2.7</th>\n",
       "      <th>numpy</th>\n",
       "      <th>list</th>\n",
       "      <th>matplotlib</th>\n",
       "      <th>dictionary</th>\n",
       "      <th>regex</th>\n",
       "      <th>flask</th>\n",
       "      <th>...</th>\n",
       "      <th>output</th>\n",
       "      <th>urllib3</th>\n",
       "      <th>login</th>\n",
       "      <th>get</th>\n",
       "      <th>https</th>\n",
       "      <th>httprequest</th>\n",
       "      <th>http-post</th>\n",
       "      <th>http-headers</th>\n",
       "      <th>multipartform-data</th>\n",
       "      <th>grequests</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>870.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>arrays</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9475.0</td>\n",
       "      <td>4291.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1931.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beautifulsoup</th>\n",
       "      <td>123.0</td>\n",
       "      <td>1411.0</td>\n",
       "      <td>168.0</td>\n",
       "      <td>1188.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>75.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49.0</td>\n",
       "      <td>421.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c++</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>class</th>\n",
       "      <td>NaN</td>\n",
       "      <td>971.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>572.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>981.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>468.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 863 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               django  python-3.x  pandas  python-2.7   numpy    list  \\\n",
       "algorithm         NaN         NaN     NaN         NaN     NaN   870.0   \n",
       "arrays            NaN         NaN     NaN         NaN  9475.0  4291.0   \n",
       "beautifulsoup   123.0      1411.0   168.0      1188.0     NaN    75.0   \n",
       "c++               NaN         NaN     NaN         NaN     NaN     NaN   \n",
       "class             NaN       971.0     NaN       572.0     NaN   981.0   \n",
       "\n",
       "               matplotlib  dictionary  regex  flask    ...      output  \\\n",
       "algorithm             NaN         NaN    NaN    NaN    ...         NaN   \n",
       "arrays                NaN      1931.0    NaN    NaN    ...         NaN   \n",
       "beautifulsoup         NaN        49.0  421.0    NaN    ...         NaN   \n",
       "c++                   NaN         NaN    NaN    NaN    ...         NaN   \n",
       "class                 NaN       468.0    NaN    NaN    ...         NaN   \n",
       "\n",
       "               urllib3  login  get  https  httprequest  http-post  \\\n",
       "algorithm          NaN    NaN  NaN    NaN          NaN        NaN   \n",
       "arrays             NaN    NaN  NaN    NaN          NaN        NaN   \n",
       "beautifulsoup      NaN    NaN  NaN    NaN          NaN        NaN   \n",
       "c++                NaN    NaN  NaN    NaN          NaN        NaN   \n",
       "class              NaN    NaN  NaN    NaN          NaN        NaN   \n",
       "\n",
       "               http-headers  multipartform-data  grequests  \n",
       "algorithm               NaN                 NaN        NaN  \n",
       "arrays                  NaN                 NaN        NaN  \n",
       "beautifulsoup           NaN                 NaN        NaN  \n",
       "c++                     NaN                 NaN        NaN  \n",
       "class                   NaN                 NaN        NaN  \n",
       "\n",
       "[5 rows x 863 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame.from_dict(DATA, orient='index')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, replace all NaNs with zero:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>django</th>\n",
       "      <th>python-3.x</th>\n",
       "      <th>pandas</th>\n",
       "      <th>python-2.7</th>\n",
       "      <th>numpy</th>\n",
       "      <th>list</th>\n",
       "      <th>matplotlib</th>\n",
       "      <th>dictionary</th>\n",
       "      <th>regex</th>\n",
       "      <th>flask</th>\n",
       "      <th>...</th>\n",
       "      <th>output</th>\n",
       "      <th>urllib3</th>\n",
       "      <th>login</th>\n",
       "      <th>get</th>\n",
       "      <th>https</th>\n",
       "      <th>httprequest</th>\n",
       "      <th>http-post</th>\n",
       "      <th>http-headers</th>\n",
       "      <th>multipartform-data</th>\n",
       "      <th>grequests</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>algorithm</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>870.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>arrays</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9475.0</td>\n",
       "      <td>4291.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1931.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>beautifulsoup</th>\n",
       "      <td>123.0</td>\n",
       "      <td>1411.0</td>\n",
       "      <td>168.0</td>\n",
       "      <td>1188.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>421.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c++</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>class</th>\n",
       "      <td>0.0</td>\n",
       "      <td>971.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>572.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>981.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>468.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 863 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               django  python-3.x  pandas  python-2.7   numpy    list  \\\n",
       "algorithm         0.0         0.0     0.0         0.0     0.0   870.0   \n",
       "arrays            0.0         0.0     0.0         0.0  9475.0  4291.0   \n",
       "beautifulsoup   123.0      1411.0   168.0      1188.0     0.0    75.0   \n",
       "c++               0.0         0.0     0.0         0.0     0.0     0.0   \n",
       "class             0.0       971.0     0.0       572.0     0.0   981.0   \n",
       "\n",
       "               matplotlib  dictionary  regex  flask    ...      output  \\\n",
       "algorithm             0.0         0.0    0.0    0.0    ...         0.0   \n",
       "arrays                0.0      1931.0    0.0    0.0    ...         0.0   \n",
       "beautifulsoup         0.0        49.0  421.0    0.0    ...         0.0   \n",
       "c++                   0.0         0.0    0.0    0.0    ...         0.0   \n",
       "class                 0.0       468.0    0.0    0.0    ...         0.0   \n",
       "\n",
       "               urllib3  login  get  https  httprequest  http-post  \\\n",
       "algorithm          0.0    0.0  0.0    0.0          0.0        0.0   \n",
       "arrays             0.0    0.0  0.0    0.0          0.0        0.0   \n",
       "beautifulsoup      0.0    0.0  0.0    0.0          0.0        0.0   \n",
       "c++                0.0    0.0  0.0    0.0          0.0        0.0   \n",
       "class              0.0    0.0  0.0    0.0          0.0        0.0   \n",
       "\n",
       "               http-headers  multipartform-data  grequests  \n",
       "algorithm               0.0                 0.0        0.0  \n",
       "arrays                  0.0                 0.0        0.0  \n",
       "beautifulsoup           0.0                 0.0        0.0  \n",
       "c++                     0.0                 0.0        0.0  \n",
       "class                   0.0                 0.0        0.0  \n",
       "\n",
       "[5 rows x 863 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_pickle(\"json_dataframe_notebook_05.pickle\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}