Machine_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine learning to predict ratings of app"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a continuation of the project using the Google Play Store data. I've previously cleaned and preprocessed the data before outputting them into a separate csv."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalising data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the columns have huge difference between the largest and smallest number, it would be nice to normalise them before running machine learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>App</th>\n",
       "      <th>Category</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Reviews</th>\n",
       "      <th>Size</th>\n",
       "      <th>Installs</th>\n",
       "      <th>Type</th>\n",
       "      <th>Price</th>\n",
       "      <th>Content Rating</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Last Updated</th>\n",
       "      <th>Current Ver</th>\n",
       "      <th>Android Ver</th>\n",
       "      <th>Category_num</th>\n",
       "      <th>Types_num</th>\n",
       "      <th>Content_num</th>\n",
       "      <th>Ratings_imp</th>\n",
       "      <th>Size_Mb</th>\n",
       "      <th>Installs_log</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Photo Editor &amp; Candy Camera &amp; Grid &amp; ScrapBook</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.1</td>\n",
       "      <td>159</td>\n",
       "      <td>19M</td>\n",
       "      <td>10000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design</td>\n",
       "      <td>2018-01-07</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>4.0.3 and up</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.1</td>\n",
       "      <td>19.0</td>\n",
       "      <td>9.210440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Coloring book moana</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>3.9</td>\n",
       "      <td>967</td>\n",
       "      <td>14M</td>\n",
       "      <td>500000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design;Pretend Play</td>\n",
       "      <td>2018-01-15</td>\n",
       "      <td>2.0.0</td>\n",
       "      <td>4.0.3 and up</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3.9</td>\n",
       "      <td>14.0</td>\n",
       "      <td>13.122365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>U Launcher Lite – FREE Live Cool Themes, Hide ...</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.7</td>\n",
       "      <td>87510</td>\n",
       "      <td>8.7M</td>\n",
       "      <td>5000000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design</td>\n",
       "      <td>2018-08-01</td>\n",
       "      <td>1.2.4</td>\n",
       "      <td>4.0.3 and up</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.7</td>\n",
       "      <td>8.7</td>\n",
       "      <td>15.424949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sketch - Draw &amp; Paint</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.5</td>\n",
       "      <td>215644</td>\n",
       "      <td>25M</td>\n",
       "      <td>50000000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Teen</td>\n",
       "      <td>Art &amp; Design</td>\n",
       "      <td>2018-06-08</td>\n",
       "      <td>Varies with device</td>\n",
       "      <td>4.2 and up</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>4.5</td>\n",
       "      <td>25.0</td>\n",
       "      <td>17.727534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pixel Draw - Number Art Coloring Book</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.3</td>\n",
       "      <td>967</td>\n",
       "      <td>2.8M</td>\n",
       "      <td>100000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design;Creativity</td>\n",
       "      <td>2018-06-20</td>\n",
       "      <td>1.1</td>\n",
       "      <td>4.4 and up</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.3</td>\n",
       "      <td>2.8</td>\n",
       "      <td>11.512935</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 App        Category  Rating  \\\n",
       "0     Photo Editor & Candy Camera & Grid & ScrapBook  ART_AND_DESIGN     4.1   \n",
       "1                                Coloring book moana  ART_AND_DESIGN     3.9   \n",
       "2  U Launcher Lite – FREE Live Cool Themes, Hide ...  ART_AND_DESIGN     4.7   \n",
       "3                              Sketch - Draw & Paint  ART_AND_DESIGN     4.5   \n",
       "4              Pixel Draw - Number Art Coloring Book  ART_AND_DESIGN     4.3   \n",
       "\n",
       "   Reviews  Size  Installs  Type  Price Content Rating  \\\n",
       "0      159   19M     10000  Free    0.0       Everyone   \n",
       "1      967   14M    500000  Free    0.0       Everyone   \n",
       "2    87510  8.7M   5000000  Free    0.0       Everyone   \n",
       "3   215644   25M  50000000  Free    0.0           Teen   \n",
       "4      967  2.8M    100000  Free    0.0       Everyone   \n",
       "\n",
       "                      Genres Last Updated         Current Ver   Android Ver  \\\n",
       "0               Art & Design   2018-01-07               1.0.0  4.0.3 and up   \n",
       "1  Art & Design;Pretend Play   2018-01-15               2.0.0  4.0.3 and up   \n",
       "2               Art & Design   2018-08-01               1.2.4  4.0.3 and up   \n",
       "3               Art & Design   2018-06-08  Varies with device    4.2 and up   \n",
       "4    Art & Design;Creativity   2018-06-20                 1.1    4.4 and up   \n",
       "\n",
       "   Category_num  Types_num  Content_num  Ratings_imp  Size_Mb  Installs_log  \n",
       "0             0          0            1          4.1     19.0      9.210440  \n",
       "1             0          0            1          3.9     14.0     13.122365  \n",
       "2             0          0            1          4.7      8.7     15.424949  \n",
       "3             0          0            4          4.5     25.0     17.727534  \n",
       "4             0          0            1          4.3      2.8     11.512935  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the data\n",
    "df = pd.read_csv(\"Processed_google_df.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>App</th>\n",
       "      <th>Category</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Reviews</th>\n",
       "      <th>Size</th>\n",
       "      <th>Installs</th>\n",
       "      <th>Type</th>\n",
       "      <th>Price</th>\n",
       "      <th>Content Rating</th>\n",
       "      <th>Genres</th>\n",
       "      <th>...</th>\n",
       "      <th>Category_num</th>\n",
       "      <th>Types_num</th>\n",
       "      <th>Content_num</th>\n",
       "      <th>Ratings_imp</th>\n",
       "      <th>Size_Mb</th>\n",
       "      <th>Installs_log</th>\n",
       "      <th>Size_Mb_scaled</th>\n",
       "      <th>Installs_scaled</th>\n",
       "      <th>Price_scaled</th>\n",
       "      <th>Reviews_scaled</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Photo Editor &amp; Candy Camera &amp; Grid &amp; ScrapBook</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.1</td>\n",
       "      <td>159</td>\n",
       "      <td>19M</td>\n",
       "      <td>10000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.1</td>\n",
       "      <td>19.0</td>\n",
       "      <td>9.210440</td>\n",
       "      <td>-0.115361</td>\n",
       "      <td>-0.144491</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.118192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Coloring book moana</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>3.9</td>\n",
       "      <td>967</td>\n",
       "      <td>14M</td>\n",
       "      <td>500000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design;Pretend Play</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3.9</td>\n",
       "      <td>14.0</td>\n",
       "      <td>13.122365</td>\n",
       "      <td>-0.344851</td>\n",
       "      <td>-0.135375</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.117751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>U Launcher Lite – FREE Live Cool Themes, Hide ...</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.7</td>\n",
       "      <td>87510</td>\n",
       "      <td>8.7M</td>\n",
       "      <td>5000000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.7</td>\n",
       "      <td>8.7</td>\n",
       "      <td>15.424949</td>\n",
       "      <td>-0.588110</td>\n",
       "      <td>-0.051659</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.070489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sketch - Draw &amp; Paint</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.5</td>\n",
       "      <td>215644</td>\n",
       "      <td>25M</td>\n",
       "      <td>50000000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Teen</td>\n",
       "      <td>Art &amp; Design</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>4.5</td>\n",
       "      <td>25.0</td>\n",
       "      <td>17.727534</td>\n",
       "      <td>0.160027</td>\n",
       "      <td>0.785508</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.000513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pixel Draw - Number Art Coloring Book</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.3</td>\n",
       "      <td>967</td>\n",
       "      <td>2.8M</td>\n",
       "      <td>100000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design;Creativity</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.3</td>\n",
       "      <td>2.8</td>\n",
       "      <td>11.512935</td>\n",
       "      <td>-0.858908</td>\n",
       "      <td>-0.142817</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.117751</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 App        Category  Rating  \\\n",
       "0     Photo Editor & Candy Camera & Grid & ScrapBook  ART_AND_DESIGN     4.1   \n",
       "1                                Coloring book moana  ART_AND_DESIGN     3.9   \n",
       "2  U Launcher Lite – FREE Live Cool Themes, Hide ...  ART_AND_DESIGN     4.7   \n",
       "3                              Sketch - Draw & Paint  ART_AND_DESIGN     4.5   \n",
       "4              Pixel Draw - Number Art Coloring Book  ART_AND_DESIGN     4.3   \n",
       "\n",
       "   Reviews  Size  Installs  Type  Price Content Rating  \\\n",
       "0      159   19M     10000  Free    0.0       Everyone   \n",
       "1      967   14M    500000  Free    0.0       Everyone   \n",
       "2    87510  8.7M   5000000  Free    0.0       Everyone   \n",
       "3   215644   25M  50000000  Free    0.0           Teen   \n",
       "4      967  2.8M    100000  Free    0.0       Everyone   \n",
       "\n",
       "                      Genres  ... Category_num Types_num Content_num  \\\n",
       "0               Art & Design  ...            0         0           1   \n",
       "1  Art & Design;Pretend Play  ...            0         0           1   \n",
       "2               Art & Design  ...            0         0           1   \n",
       "3               Art & Design  ...            0         0           4   \n",
       "4    Art & Design;Creativity  ...            0         0           1   \n",
       "\n",
       "   Ratings_imp  Size_Mb  Installs_log  Size_Mb_scaled  Installs_scaled  \\\n",
       "0          4.1     19.0      9.210440       -0.115361        -0.144491   \n",
       "1          3.9     14.0     13.122365       -0.344851        -0.135375   \n",
       "2          4.7      8.7     15.424949       -0.588110        -0.051659   \n",
       "3          4.5     25.0     17.727534        0.160027         0.785508   \n",
       "4          4.3      2.8     11.512935       -0.858908        -0.142817   \n",
       "\n",
       "   Price_scaled  Reviews_scaled  \n",
       "0     -0.065232       -0.118192  \n",
       "1     -0.065232       -0.117751  \n",
       "2     -0.065232       -0.070489  \n",
       "3     -0.065232       -0.000513  \n",
       "4     -0.065232       -0.117751  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# library\n",
    "from sklearn import preprocessing\n",
    "\n",
    "# normalise\n",
    "cols_to_normalise = [\"Size_Mb\", \"Installs\", \"Price\", \"Reviews\"]\n",
    "scaler = preprocessing.StandardScaler()\n",
    "for col in cols_to_normalise:\n",
    "    new_col = col + \"_scaled\"\n",
    "    x = df[col].values.reshape(-1, 1)\n",
    "    x_scaled = scaler.fit_transform(x)\n",
    "    df[new_col] = x_scaled\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subset the training and test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load libraries\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# subset the df for the features\n",
    "variables = ['Reviews_scaled', 'Price_scaled', 'Category_num', 'Types_num', 'Content_num',\n",
    "       'Ratings_imp', 'Size_Mb_scaled', 'Installs_scaled']\n",
    "new_df = df[variables]\n",
    "x = new_df.drop(\"Ratings_imp\", axis=1)\n",
    "y = new_df[\"Ratings_imp\"]\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y, train_size=0.7, test_size=0.3, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm encoded the categorical data such as Category, Type and Content into numerical. Then, I've decided to use a logarithm scaled Installs instead of the original installs. The codes on how they are done is included in the previous notebook. I've also showed previously that the variables do not have normal distribution, so I won't be applying linear regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recursive Feature Selection: GradientBoostingRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting estimator with 7 features.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Reviews_scaled',\n",
       " 'Price_scaled',\n",
       " 'Category_num',\n",
       " 'Content_num',\n",
       " 'Installs_scaled']"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the independent variables\n",
    "ind_var = x.columns.tolist()\n",
    "\n",
    "# import libraries\n",
    "from sklearn.feature_selection import RFE\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "\n",
    "rfe_selector = RFE(estimator=GradientBoostingRegressor(), n_features_to_select=5, step=10, verbose=5)  # step=number features to remove at each iteration\n",
    "rfe_selector.fit(x, y)\n",
    "rfe_support = rfe_selector.get_support()\n",
    "rfe_feature = [var for var, boolean in zip(ind_var, rfe_support) if boolean == True]\n",
    "rfe_feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SelectFromModel: RandomForestRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Reviews_scaled</th>\n",
       "      <th>Price_scaled</th>\n",
       "      <th>Category_num</th>\n",
       "      <th>Types_num</th>\n",
       "      <th>Content_num</th>\n",
       "      <th>Size_Mb_scaled</th>\n",
       "      <th>Installs_scaled</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Reviews_scaled</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.007597</td>\n",
       "      <td>0.017293</td>\n",
       "      <td>-0.033076</td>\n",
       "      <td>0.055630</td>\n",
       "      <td>0.176755</td>\n",
       "      <td>0.625165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Price_scaled</th>\n",
       "      <td>-0.007597</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.013770</td>\n",
       "      <td>0.223868</td>\n",
       "      <td>-0.014485</td>\n",
       "      <td>-0.024251</td>\n",
       "      <td>-0.009404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Category_num</th>\n",
       "      <td>0.017293</td>\n",
       "      <td>-0.013770</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.016989</td>\n",
       "      <td>-0.093881</td>\n",
       "      <td>-0.093615</td>\n",
       "      <td>0.031655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Types_num</th>\n",
       "      <td>-0.033076</td>\n",
       "      <td>0.223868</td>\n",
       "      <td>0.016989</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.041921</td>\n",
       "      <td>-0.030307</td>\n",
       "      <td>-0.041746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Content_num</th>\n",
       "      <td>0.055630</td>\n",
       "      <td>-0.014485</td>\n",
       "      <td>-0.093881</td>\n",
       "      <td>-0.041921</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.175586</td>\n",
       "      <td>0.049814</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Size_Mb_scaled</th>\n",
       "      <td>0.176755</td>\n",
       "      <td>-0.024251</td>\n",
       "      <td>-0.093615</td>\n",
       "      <td>-0.030307</td>\n",
       "      <td>0.175586</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.132662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Installs_scaled</th>\n",
       "      <td>0.625165</td>\n",
       "      <td>-0.009404</td>\n",
       "      <td>0.031655</td>\n",
       "      <td>-0.041746</td>\n",
       "      <td>0.049814</td>\n",
       "      <td>0.132662</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Reviews_scaled  Price_scaled  Category_num  Types_num  \\\n",
       "Reviews_scaled         1.000000     -0.007597      0.017293  -0.033076   \n",
       "Price_scaled          -0.007597      1.000000     -0.013770   0.223868   \n",
       "Category_num           0.017293     -0.013770      1.000000   0.016989   \n",
       "Types_num             -0.033076      0.223868      0.016989   1.000000   \n",
       "Content_num            0.055630     -0.014485     -0.093881  -0.041921   \n",
       "Size_Mb_scaled         0.176755     -0.024251     -0.093615  -0.030307   \n",
       "Installs_scaled        0.625165     -0.009404      0.031655  -0.041746   \n",
       "\n",
       "                 Content_num  Size_Mb_scaled  Installs_scaled  \n",
       "Reviews_scaled      0.055630        0.176755         0.625165  \n",
       "Price_scaled       -0.014485       -0.024251        -0.009404  \n",
       "Category_num       -0.093881       -0.093615         0.031655  \n",
       "Types_num          -0.041921       -0.030307        -0.041746  \n",
       "Content_num         1.000000        0.175586         0.049814  \n",
       "Size_Mb_scaled      0.175586        1.000000         0.132662  \n",
       "Installs_scaled     0.049814        0.132662         1.000000  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# correlation matrix\n",
    "x.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Reviews_scaled', 'Category_num', 'Size_Mb_scaled', 'Installs_scaled']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# there are no variables that have high correlation with each other\n",
    "\n",
    "# libraries\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "rf_selector = SelectFromModel(RandomForestRegressor(), max_features=5)\n",
    "rf_selector.fit(x, y)\n",
    "\n",
    "rf_support = rf_selector.get_support()\n",
    "rf_features = [var for var, boolean in zip(ind_var, rf_support) if boolean == True]\n",
    "rf_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SelectFromModel: XGBoostRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Price_scaled', 'Category_num', 'Content_num', 'Installs_scaled']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# library\n",
    "from xgboost import XGBRegressor\n",
    "model = XGBRegressor(objective='reg:squarederror')\n",
    "\n",
    "xg_selector = SelectFromModel(model, max_features=5)\n",
    "xg_selector.fit(x, y)\n",
    "\n",
    "xg_support = xg_selector.get_support()\n",
    "xg_features = [var for var, boolean in zip(ind_var, xg_support) if boolean == True]\n",
    "xg_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mutual Information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mutual information measures the contribution of a variable towards another variables. It allows one to non-linear relationship"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Reviews_scaled',\n",
       " 'Category_num',\n",
       " 'Content_num',\n",
       " 'Size_Mb_scaled',\n",
       " 'Installs_scaled']"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# library\n",
    "from sklearn.feature_selection import mutual_info_regression\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "\n",
    "selector = SelectKBest(mutual_info_regression, k=5)\n",
    "new_training = selector.fit_transform(x, y)\n",
    "mi_mask = selector.get_support()\n",
    "mi_features = [var for var, boolean in zip(ind_var, mi_mask) if boolean == True]\n",
    "mi_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combining all of them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Features</th>\n",
       "      <th>GBRegressor</th>\n",
       "      <th>RFRegressor</th>\n",
       "      <th>XGBRegressor</th>\n",
       "      <th>MI</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Installs_scaled</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Category_num</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Reviews_scaled</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Content_num</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Size_Mb_scaled</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Price_scaled</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Types_num</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Features  GBRegressor  RFRegressor  XGBRegressor     MI  Total\n",
       "0  Installs_scaled         True         True          True   True      4\n",
       "1     Category_num         True         True          True   True      4\n",
       "2   Reviews_scaled         True         True         False   True      3\n",
       "3      Content_num         True        False          True   True      3\n",
       "4   Size_Mb_scaled        False         True         False   True      2\n",
       "5     Price_scaled         True        False          True  False      2\n",
       "6        Types_num        False        False         False  False      0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dict = {\n",
    "    \"Features\": ind_var,\n",
    "    \"GBRegressor\": rfe_support,\n",
    "    \"RFRegressor\": rf_support,\n",
    "    \"XGBRegressor\": xg_support,\n",
    "    \"MI\": mi_mask\n",
    "}\n",
    "feature_selection_df = pd.DataFrame(data_dict)\n",
    "feature_selection_df[\"Total\"] = np.sum(feature_selection_df, axis=1)\n",
    "\n",
    "# sort\n",
    "feature_selection_df = feature_selection_df.sort_values([\"Total\", \"Features\"], ascending=False)\n",
    "feature_selection_df=feature_selection_df.reset_index(drop=True)\n",
    "feature_selection_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that out of the 4 different feature selection methods, Installs and Category types are highly important contributors to predicting the ratings. \n",
    "\n",
    "There is one caveat to the method I was using, however. I applied LabelEncoding to turn them into numerical, when training, regression will think of them as ordinal, ie 1 is better than 2. However, we have 33 levels in Category alone, which is a bit much to use one-hot encoder.We'll experiment with one hot encoding afterwards to see if there's a difference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I decided to use Random Forest Regressor and XGBoost Regressor. RF is a popular algorithm given that it's fast and general give pretty accurate results. It does have limitations, namely, it doesn't extrapolate well. As for XGBoost, I just want to try it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## With all variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(bootstrap=True, ccp_alpha=0.0, criterion='mse',\n",
       "                      max_depth=None, max_features='auto', max_leaf_nodes=100,\n",
       "                      max_samples=None, min_impurity_decrease=0.0,\n",
       "                      min_impurity_split=None, min_samples_leaf=1,\n",
       "                      min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "                      n_estimators=100, n_jobs=None, oob_score=False,\n",
       "                      random_state=0, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# model\n",
    "rf_model = RandomForestRegressor(max_leaf_nodes=100, random_state=0)\n",
    "rf_model.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r2 is 0.2663787790090971, mae is 0.35733750397542874 and msr is 0.2500778304548192.\n"
     ]
    }
   ],
   "source": [
    "# libraries\n",
    "from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error\n",
    "\n",
    "rf_pred = rf_model.predict(x_test)\n",
    "r2_rf = r2_score(y_test, rf_pred)\n",
    "msr_rf = mean_squared_error(y_test, rf_pred)\n",
    "mae_rf = mean_absolute_error(y_test, rf_pred)\n",
    "r2_rf, msr_rf, mae_rf\n",
    "print(f\"r2 is {r2_rf}, mae is {mae_rf} and msr is {msr_rf}.\")  # r2 is 0.11768667951622525, mae is 0.3848103252802534 and msr is 0.3007642018178562."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.45126532160548605"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_model.score(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Leaf nodes is 5, r2 is 0.0957725407727712 and mae is 0.40031693315129185.\n",
      "Leaf nodes is 10, r2 is 0.14661349629197118 and mae is 0.3913406641186324.\n",
      "Leaf nodes is 100, r2 is 0.2663787790090971 and mae is 0.35733750397542874.\n",
      "Leaf nodes is 500, r2 is 0.28847541086249673 and mae is 0.3445534238296447.\n",
      "Leaf nodes is 1000, r2 is 0.2792830150910389 and mae is 0.3453183903148569.\n",
      "Leaf nodes is 5000, r2 is 0.2698598875639675 and mae is 0.3469667130192774.\n",
      "Leaf nodes is 10000, r2 is 0.2698598875639675 and mae is 0.3469667130192774.\n",
      "Leaf nodes is None, r2 is 0.26899070280145243 and mae is 0.34781902623765165.\n"
     ]
    }
   ],
   "source": [
    "# experiment with different max_leaf_node\n",
    "leaf_nodes = [5,10,100, 500, 1000, 5000, 10000, None]\n",
    "for nodes in leaf_nodes:\n",
    "    tmp_model = RandomForestRegressor(max_leaf_nodes=nodes, random_state=0)\n",
    "    tmp_model.fit(x_train, y_train)\n",
    "    tmp_pred = tmp_model.predict(x_test)\n",
    "    tmp_r2 = r2_score(y_test, tmp_pred)\n",
    "    tmp_mae = mean_absolute_error(y_test, tmp_pred)\n",
    "    print(f\"Leaf nodes is {nodes}, r2 is {tmp_r2} and mae is {tmp_mae}.\")  # the best score achieved is when max_leaf_nodes is 500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XGBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I understand that Random Forest and XGB have the similarity that both uses decision trees at the basic level but there are differences between the both of them such how the trees are built and combined. https://www.datasciencecentral.com/profiles/blogs/decision-tree-vs-random-forest-vs-boosted-trees-explained"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r2 is 0.23722900655660106, mae is 0.3621262373268132 and msr is 0.26001444576063826.\n"
     ]
    }
   ],
   "source": [
    "# libraries \n",
    "import xgboost as xg\n",
    "\n",
    "xg_model = xg.XGBRegressor(objective=\"reg:squaredlogerror\", random_state=0, eta=0.25, max_depth=8)\n",
    "xg_model.fit(x_train, y_train)\n",
    "\n",
    "xg_pred = xg_model.predict(x_test)\n",
    "xg_r2 = r2_score(y_test, xg_pred)\n",
    "xg_msr = mean_squared_error(y_test, xg_pred)\n",
    "xg_mae = mean_absolute_error(y_test, xg_pred)\n",
    "print(f\"r2 is {xg_r2}, mae is {xg_mae} and msr is {xg_msr}.\") \n",
    "# without any tuning, the xg model performs poorly compared to random forest\n",
    "# r2 is 0.2371537048975535, mae is 0.36212612609771766 and msr is 0.2600401146957583."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using only 4 variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using only the 4 variables from feature selection, let's test and see if the models fare better!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# subset\n",
    "new_var = [\"Category_num\", \"Content_num\", \"Reviews_scaled\", \"Installs_scaled\", \"Ratings_imp\"]\n",
    "fs_df = df[new_var]  # fs is feature selection\n",
    "fs_x = fs_df.drop(\"Ratings_imp\", axis=1)\n",
    "fs_y = fs_df[\"Ratings_imp\"]\n",
    "\n",
    "# train_test_split\n",
    "fs_x_train, fs_x_test, fs_y_train, fs_y_test = train_test_split(fs_x,fs_y, train_size=0.7, test_size=0.3, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RandomForestRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForestRegressor: r2 is 0.27535109094579546, mae is 0.3521197209149075 and mse is 0.24701933618135377.\n"
     ]
    }
   ],
   "source": [
    "fs_rf_model = RandomForestRegressor(max_leaf_nodes=100, random_state=0)\n",
    "fs_rf_model.fit(fs_x_train, fs_y_train)\n",
    "\n",
    "fs_rf_pred = fs_rf_model.predict(fs_x_test)\n",
    "fs_r2_rf = r2_score(fs_y_test, fs_rf_pred)\n",
    "fs_msr_rf = mean_squared_error(fs_y_test, fs_rf_pred)\n",
    "fs_mae_rf = mean_absolute_error(fs_y_test, fs_rf_pred)\n",
    "rf_metrics = [fs_r2_rf, fs_mae_rf, fs_msr_rf]\n",
    "print(f\"RandomForestRegressor: r2 is {fs_r2_rf}, mae is {fs_mae_rf} and mse is {fs_msr_rf}.\")  \n",
    "\n",
    "# when compared to using all variables, using only 4 variables fare better in that the r2 is higher, msr is smaller and mae is a bit smaller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "XGBoost: r2 is 0.24815473371158425, mae is 0.3584163443848441 and mse is 0.2562900685685922.\n"
     ]
    }
   ],
   "source": [
    "fs_xg_model = xg.XGBRegressor(objective=\"reg:squaredlogerror\", random_state=0, eta=0.25, max_depth=8)\n",
    "fs_xg_model.fit(fs_x_train, fs_y_train)\n",
    "\n",
    "fs_xg_pred = fs_xg_model.predict(fs_x_test)\n",
    "fs_r2_xg = r2_score(fs_y_test, fs_xg_pred)\n",
    "fs_msr_xg = mean_squared_error(fs_y_test, fs_xg_pred)\n",
    "fs_mae_xg = mean_absolute_error(fs_y_test, fs_xg_pred)\n",
    "xg_metrics = [fs_r2_xg, fs_mae_xg, fs_msr_xg]\n",
    "print(f\"XGBoost: r2 is {fs_r2_xg}, mae is {fs_mae_xg} and mse is {fs_msr_xg}.\") \n",
    "# when using 4 variables, the metrics is better than when using all variables but still not better than random forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, it seem Random Forest performs better than XGBoost in this case but still it's still not a great model. Somebody used KNN and got 94% accuracy. I'm going to give it a go."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNNReressor: r2 is 0.2101375248796329, mae is 0.3560316244372145 and msr is 0.2692494280208729.\n"
     ]
    }
   ],
   "source": [
    "# library\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "knn_fs_model = KNeighborsRegressor()\n",
    "\n",
    "knn_fs_model.fit(fs_x_train, fs_y_train)\n",
    "knn_pred = knn_fs_model.predict(fs_x_test)\n",
    "\n",
    "knn_r2 = r2_score(fs_y_test, knn_pred)\n",
    "knn_msr = mean_squared_error(fs_y_test, knn_pred)\n",
    "knn_mae = mean_absolute_error(fs_y_test, knn_pred)\n",
    "knn_metrics = [knn_r2, knn_mae, knn_msr]\n",
    "print(f\"KNNReressor: r2 is {knn_r2}, mae is {knn_mae} and msr is {knn_msr}.\") \n",
    "# r2 is 0.1331057815713581, mae is 0.3856934108909264 and msr is 0.2955081166894833.\n",
    "# nope, this doesn't look like the best model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support Vector Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SupportVectorRegression: r2 is -0.025321337538159572, mae is 0.3995576515715841 and msr is 0.3595575781722654.\n"
     ]
    }
   ],
   "source": [
    "# I'm curious about it so decided to give it a try\n",
    "# library\n",
    "from sklearn.svm import SVR\n",
    "\n",
    "# data\n",
    "variables = [\"Category_num\", \"Content_num\", \"Reviews_scaled\", \"Installs_scaled\", \"Ratings_imp\"]\n",
    "new_df = df[variables]\n",
    "x = new_df.drop(\"Ratings_imp\", axis=1)\n",
    "y = new_df[\"Ratings_imp\"]\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y, train_size=0.9, test_size=0.1, random_state=8)\n",
    "\n",
    "# model\n",
    "svr_model = SVR(kernel=\"poly\")  # tried all the options, poly seems to the best in this data\n",
    "svr_model.fit(x_train, y_train)\n",
    "svr_pred = svr_model.predict(x_test)\n",
    "svr_r2 = r2_score(y_test, svr_pred)\n",
    "svr_mse = mean_squared_error(y_test, svr_pred)\n",
    "svr_mae = mean_absolute_error(y_test, svr_pred)\n",
    "svr_metrics = [svr_r2, svr_mae, svr_mse]\n",
    "print(f\"SupportVectorRegression: r2 is {svr_r2}, mae is {svr_mae} and msr is {svr_mse}.\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metrics</th>\n",
       "      <th>RFRegressor</th>\n",
       "      <th>XGBRegressor</th>\n",
       "      <th>KNNRegressor</th>\n",
       "      <th>SVRgressor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>R2</td>\n",
       "      <td>0.275351</td>\n",
       "      <td>0.248155</td>\n",
       "      <td>0.210138</td>\n",
       "      <td>-0.025321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mean Absolute Error</td>\n",
       "      <td>0.352120</td>\n",
       "      <td>0.358416</td>\n",
       "      <td>0.356032</td>\n",
       "      <td>0.399558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Mean Squared Error</td>\n",
       "      <td>0.247019</td>\n",
       "      <td>0.256290</td>\n",
       "      <td>0.269249</td>\n",
       "      <td>0.359558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Metrics  RFRegressor  XGBRegressor  KNNRegressor  SVRgressor\n",
       "0                   R2     0.275351      0.248155      0.210138   -0.025321\n",
       "1  Mean Absolute Error     0.352120      0.358416      0.356032    0.399558\n",
       "2   Mean Squared Error     0.247019      0.256290      0.269249    0.359558"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# gather all information\n",
    "data_dict = {\n",
    "    \"Metrics\" : [\"R2\", \"Mean Absolute Error\", \"Mean Squared Error\"],\n",
    "    \"RFRegressor\": rf_metrics,\n",
    "    \"XGBRegressor\": xg_metrics,\n",
    "    \"KNNRegressor\": knn_metrics,\n",
    "    \"SVRgressor\": svr_metrics\n",
    "}\n",
    "\n",
    "model_df = pd.DataFrame(data_dict)\n",
    "model_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using only variables specify per model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is I use only the variables output by each model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "variables = ['Category_num','Ratings_imp','Installs_scaled', 'Reviews_scaled', 'Size_Mb_scaled']\n",
    "new_df = df[variables]\n",
    "x = new_df.drop(\"Ratings_imp\", axis=1)\n",
    "y = new_df[\"Ratings_imp\"]\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y, train_size=0.7, test_size=0.3, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r2 is 0.1935692731259585, mae is 0.3641288622499256 and msr is 0.27489723690975937.\n"
     ]
    }
   ],
   "source": [
    "# random forest\n",
    "rf_model = RandomForestRegressor(random_state=0)\n",
    "rf_model.fit(x_train, y_train)\n",
    "\n",
    "rf_pred = rf_model.predict(x_test)\n",
    "r2_rf = r2_score(y_test, rf_pred)\n",
    "msr_rf = mean_squared_error(y_test, rf_pred)\n",
    "mae_rf = mean_absolute_error(y_test, rf_pred)\n",
    "r2_rf, msr_rf, mae_rf\n",
    "print(f\"r2 is {r2_rf}, mae is {mae_rf} and msr is {msr_rf}.\")  # nope not any better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Features</th>\n",
       "      <th>GBRegressor</th>\n",
       "      <th>RFRegressor</th>\n",
       "      <th>XGBRegressor</th>\n",
       "      <th>MI</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Installs_scaled</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Category_num</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Reviews_scaled</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Price_scaled</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Size_Mb_scaled</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Content_num</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Types_num</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Features  GBRegressor  RFRegressor  XGBRegressor     MI  Total\n",
       "0  Installs_scaled         True         True          True   True      4\n",
       "1     Category_num         True         True          True   True      4\n",
       "2   Reviews_scaled         True         True         False   True      3\n",
       "3     Price_scaled         True        False          True   True      3\n",
       "4   Size_Mb_scaled        False         True         False   True      2\n",
       "5      Content_num         True        False          True  False      2\n",
       "6        Types_num        False        False         False  False      0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_selection_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r2 is 0.16795558792457632, mae is 0.38504045950656945 and msr is 0.28362846583531115.\n"
     ]
    }
   ],
   "source": [
    "# xg\n",
    "variables = ['Category_num','Ratings_imp', 'Installs_scaled', 'Content_num', 'Price_scaled']\n",
    "new_df = df[variables]\n",
    "x = new_df.drop(\"Ratings_imp\", axis=1)\n",
    "y = new_df[\"Ratings_imp\"]\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y, train_size=0.7, test_size=0.3, random_state=0)\n",
    "\n",
    "xg_model = xg.XGBRegressor(objective=\"reg:squaredlogerror\", random_state=0, eta=0.25, max_depth=8)\n",
    "xg_model.fit(x_train, y_train)\n",
    "\n",
    "xg_pred = xg_model.predict(x_test)\n",
    "xg_r2 = r2_score(y_test, xg_pred)\n",
    "xg_msr = mean_squared_error(y_test, xg_pred)\n",
    "xg_mae = mean_absolute_error(y_test, xg_pred)\n",
    "print(f\"r2 is {xg_r2}, mae is {xg_mae} and msr is {xg_msr}.\")  # nope, not better either"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sub-final Machine Learning model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the best model that I could use at the moment is Random Forest Regressor, max_leaf_nodes at 500. The variables are the top 4. This only works when the data are not normalised."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9660, 5)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the data\n",
    "variables = [\"Category_num\", \"Content_num\", \"Reviews_scaled\", \"Installs_scaled\", \"Ratings_imp\"]\n",
    "# variables = ['Reviews', 'Price', 'Category_num', 'Types_num', 'Content_num',\n",
    "#        'Ratings_imp', 'Size_Mb', 'Installs_log']\n",
    "new_df = df[variables]\n",
    "x = new_df.drop(\"Ratings_imp\", axis=1)\n",
    "y = new_df[\"Ratings_imp\"]\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y, train_size=0.9, test_size=0.1, random_state=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r2 is 0.3950345586381413, mae is 0.32034424502864833 and msr is 0.21214803692300035.\n"
     ]
    }
   ],
   "source": [
    "rf_model = RandomForestRegressor(max_leaf_nodes=260, random_state=8, n_jobs=-1)\n",
    "rf_model.fit(x_train, y_train)\n",
    "\n",
    "rf_pred = rf_model.predict(x_test)\n",
    "r2_rf = r2_score(y_test, rf_pred)\n",
    "msr_rf = mean_squared_error(y_test, rf_pred)\n",
    "mae_rf = mean_absolute_error(y_test, rf_pred)\n",
    "print(f\"r2 is {r2_rf}, mae is {mae_rf} and msr is {msr_rf}.\")  # the best so far r2 is 0.2893709583033749, mae is 0.339268447634886 and msr is 0.242240224138613."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Leaf nodes is 10, r2 is 0.14575517445110397 and mae is 0.3820703485709907.\n",
      "Leaf nodes is 60, r2 is 0.3336184372867338 and mae is 0.34406207771861713.\n",
      "Leaf nodes is 110, r2 is 0.3724521929586293 and mae is 0.3317959913545915.\n",
      "Leaf nodes is 160, r2 is 0.39055920779465714 and mae is 0.32489821370291894.\n",
      "Leaf nodes is 210, r2 is 0.39618254921764406 and mae is 0.3220245918320109.\n",
      "Leaf nodes is 260, r2 is 0.3971318851179779 and mae is 0.3204367515399854.\n"
     ]
    }
   ],
   "source": [
    "leaf_nodes = np.arange(10, 1000, 50)\n",
    "best = 0\n",
    "for nodes in leaf_nodes:\n",
    "    tmp_model = RandomForestRegressor(max_leaf_nodes=nodes, random_state=8, n_jobs=-1)\n",
    "    tmp_model.set_params(n_estimators=80)\n",
    "    tmp_model.fit(x_train, y_train)\n",
    "    tmp_pred = tmp_model.predict(x_test)\n",
    "    tmp_r2 = r2_score(y_test, tmp_pred)\n",
    "    tmp_mae = mean_absolute_error(y_test, tmp_pred)\n",
    "    if tmp_r2 > best:\n",
    "        print(f\"Leaf nodes is {nodes}, r2 is {tmp_r2} and mae is {tmp_mae}.\")\n",
    "        best = tmp_r2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "estimator is 10,r2 is 0.3677699546153661, mae is 0.3255453376707456 and mse is 0.22170913219464733.\n",
      "estimator is 20,r2 is 0.37738119399287506, mae is 0.32272443882033136 and mse is 0.2183386825343403.\n",
      "estimator is 30,r2 is 0.383127869015377, mae is 0.32280750539288977 and mse is 0.21632345035493883.\n",
      "estimator is 40,r2 is 0.38667106341056756, mae is 0.32201374213955425 and mse is 0.21508093023067504.\n",
      "estimator is 50,r2 is 0.39053522518870676, mae is 0.3217662699252988 and mse is 0.21372585392459761.\n",
      "estimator is 60,r2 is 0.39551137620779897, mae is 0.321793590724716 and mse is 0.21198082751820269.\n",
      "estimator is 70,r2 is 0.3967557764855585, mae is 0.3208985655826423 and mse is 0.21154444378778853.\n",
      "estimator is 80,r2 is 0.39713188511797815, mae is 0.3204367515399854 and mse is 0.21141255078600318.\n"
     ]
    }
   ],
   "source": [
    "estimators = np.arange(10, 500, 10)\n",
    "best = 0\n",
    "for n in estimators:\n",
    "    rf_model.set_params(n_estimators=n)\n",
    "    rf_model.fit(x_train, y_train)\n",
    "    rf_pred = rf_model.predict(x_test)\n",
    "    r2_rf = r2_score(y_test, rf_pred)\n",
    "    mse_rf = mean_squared_error(y_test, rf_pred)\n",
    "    mae_rf = mean_absolute_error(y_test, rf_pred)\n",
    "    if best < r2_rf:\n",
    "        print(f\"estimator is {n},r2 is {r2_rf}, mae is {mae_rf} and mse is {mse_rf}.\")\n",
    "        best = r2_rf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the best I can get with Random Forest for now is max_leaf_nodes at 260, n_estimators at 80 with 4 variables:\n",
    "- Installs_log\n",
    "- Category_num\n",
    "- Reviews_scaled\n",
    "- Content_num"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r2 is 0.3971318851179779, mae is 0.3204367515399854 and msr is 0.21141255078600324.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.3971318851179779"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the conclusive machine learning model for this round\n",
    "variables = [\"Category_num\", \"Content_num\", \"Reviews_scaled\", \"Installs_scaled\", \"Ratings_imp\"]\n",
    "new_df = df[variables]\n",
    "x = new_df.drop(\"Ratings_imp\", axis=1)\n",
    "y = new_df[\"Ratings_imp\"]\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y, train_size=0.9, test_size=0.1, random_state=8)\n",
    "\n",
    "rf_model = RandomForestRegressor(max_leaf_nodes=260, random_state=8, n_jobs=-1, n_estimators=80)\n",
    "rf_model.fit(x_train, y_train)\n",
    "rf_pred = rf_model.predict(x_test)\n",
    "r2_rf = r2_score(y_test, rf_pred)\n",
    "msr_rf = mean_squared_error(y_test, rf_pred)\n",
    "mae_rf = mean_absolute_error(y_test, rf_pred)\n",
    "print(f\"r2 is {r2_rf}, mae is {mae_rf} and msr is {msr_rf}.\") \n",
    "rf_model.score(x_test, y_test)  # the accuracy of the model on testing set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.45979620571075097"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import math\n",
    "math.sqrt(msr_rf)  # rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine learning on data with dummy variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>App</th>\n",
       "      <th>Category</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Reviews</th>\n",
       "      <th>Size</th>\n",
       "      <th>Installs</th>\n",
       "      <th>Type</th>\n",
       "      <th>Price</th>\n",
       "      <th>Content Rating</th>\n",
       "      <th>Genres</th>\n",
       "      <th>...</th>\n",
       "      <th>Category_num</th>\n",
       "      <th>Types_num</th>\n",
       "      <th>Content_num</th>\n",
       "      <th>Ratings_imp</th>\n",
       "      <th>Size_Mb</th>\n",
       "      <th>Installs_log</th>\n",
       "      <th>Size_Mb_scaled</th>\n",
       "      <th>Installs_scaled</th>\n",
       "      <th>Price_scaled</th>\n",
       "      <th>Reviews_scaled</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Photo Editor &amp; Candy Camera &amp; Grid &amp; ScrapBook</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.1</td>\n",
       "      <td>159</td>\n",
       "      <td>19M</td>\n",
       "      <td>10000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.1</td>\n",
       "      <td>19.0</td>\n",
       "      <td>9.210440</td>\n",
       "      <td>-0.115361</td>\n",
       "      <td>-0.144491</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.118192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Coloring book moana</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>3.9</td>\n",
       "      <td>967</td>\n",
       "      <td>14M</td>\n",
       "      <td>500000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design;Pretend Play</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3.9</td>\n",
       "      <td>14.0</td>\n",
       "      <td>13.122365</td>\n",
       "      <td>-0.344851</td>\n",
       "      <td>-0.135375</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.117751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>U Launcher Lite – FREE Live Cool Themes, Hide ...</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.7</td>\n",
       "      <td>87510</td>\n",
       "      <td>8.7M</td>\n",
       "      <td>5000000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.7</td>\n",
       "      <td>8.7</td>\n",
       "      <td>15.424949</td>\n",
       "      <td>-0.588110</td>\n",
       "      <td>-0.051659</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.070489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sketch - Draw &amp; Paint</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.5</td>\n",
       "      <td>215644</td>\n",
       "      <td>25M</td>\n",
       "      <td>50000000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Teen</td>\n",
       "      <td>Art &amp; Design</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>4.5</td>\n",
       "      <td>25.0</td>\n",
       "      <td>17.727534</td>\n",
       "      <td>0.160027</td>\n",
       "      <td>0.785508</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.000513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pixel Draw - Number Art Coloring Book</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>4.3</td>\n",
       "      <td>967</td>\n",
       "      <td>2.8M</td>\n",
       "      <td>100000</td>\n",
       "      <td>Free</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Everyone</td>\n",
       "      <td>Art &amp; Design;Creativity</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.3</td>\n",
       "      <td>2.8</td>\n",
       "      <td>11.512935</td>\n",
       "      <td>-0.858908</td>\n",
       "      <td>-0.142817</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>-0.117751</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 App        Category  Rating  \\\n",
       "0     Photo Editor & Candy Camera & Grid & ScrapBook  ART_AND_DESIGN     4.1   \n",
       "1                                Coloring book moana  ART_AND_DESIGN     3.9   \n",
       "2  U Launcher Lite – FREE Live Cool Themes, Hide ...  ART_AND_DESIGN     4.7   \n",
       "3                              Sketch - Draw & Paint  ART_AND_DESIGN     4.5   \n",
       "4              Pixel Draw - Number Art Coloring Book  ART_AND_DESIGN     4.3   \n",
       "\n",
       "   Reviews  Size  Installs  Type  Price Content Rating  \\\n",
       "0      159   19M     10000  Free    0.0       Everyone   \n",
       "1      967   14M    500000  Free    0.0       Everyone   \n",
       "2    87510  8.7M   5000000  Free    0.0       Everyone   \n",
       "3   215644   25M  50000000  Free    0.0           Teen   \n",
       "4      967  2.8M    100000  Free    0.0       Everyone   \n",
       "\n",
       "                      Genres  ... Category_num Types_num Content_num  \\\n",
       "0               Art & Design  ...            0         0           1   \n",
       "1  Art & Design;Pretend Play  ...            0         0           1   \n",
       "2               Art & Design  ...            0         0           1   \n",
       "3               Art & Design  ...            0         0           4   \n",
       "4    Art & Design;Creativity  ...            0         0           1   \n",
       "\n",
       "   Ratings_imp  Size_Mb  Installs_log  Size_Mb_scaled  Installs_scaled  \\\n",
       "0          4.1     19.0      9.210440       -0.115361        -0.144491   \n",
       "1          3.9     14.0     13.122365       -0.344851        -0.135375   \n",
       "2          4.7      8.7     15.424949       -0.588110        -0.051659   \n",
       "3          4.5     25.0     17.727534        0.160027         0.785508   \n",
       "4          4.3      2.8     11.512935       -0.858908        -0.142817   \n",
       "\n",
       "   Price_scaled  Reviews_scaled  \n",
       "0     -0.065232       -0.118192  \n",
       "1     -0.065232       -0.117751  \n",
       "2     -0.065232       -0.070489  \n",
       "3     -0.065232       -0.000513  \n",
       "4     -0.065232       -0.117751  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Reviews_scaled</th>\n",
       "      <th>Category</th>\n",
       "      <th>Price_scaled</th>\n",
       "      <th>Types_num</th>\n",
       "      <th>Content_num</th>\n",
       "      <th>Ratings_imp</th>\n",
       "      <th>Size_Mb_scaled</th>\n",
       "      <th>Installs_scaled</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.118192</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.1</td>\n",
       "      <td>-0.115361</td>\n",
       "      <td>-0.144491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.117751</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3.9</td>\n",
       "      <td>-0.344851</td>\n",
       "      <td>-0.135375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.070489</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.7</td>\n",
       "      <td>-0.588110</td>\n",
       "      <td>-0.051659</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.000513</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>4.5</td>\n",
       "      <td>0.160027</td>\n",
       "      <td>0.785508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.117751</td>\n",
       "      <td>ART_AND_DESIGN</td>\n",
       "      <td>-0.065232</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.3</td>\n",
       "      <td>-0.858908</td>\n",
       "      <td>-0.142817</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Reviews_scaled        Category  Price_scaled  Types_num  Content_num  \\\n",
       "0       -0.118192  ART_AND_DESIGN     -0.065232          0            1   \n",
       "1       -0.117751  ART_AND_DESIGN     -0.065232          0            1   \n",
       "2       -0.070489  ART_AND_DESIGN     -0.065232          0            1   \n",
       "3       -0.000513  ART_AND_DESIGN     -0.065232          0            4   \n",
       "4       -0.117751  ART_AND_DESIGN     -0.065232          0            1   \n",
       "\n",
       "   Ratings_imp  Size_Mb_scaled  Installs_scaled  \n",
       "0          4.1       -0.115361        -0.144491  \n",
       "1          3.9       -0.344851        -0.135375  \n",
       "2          4.7       -0.588110        -0.051659  \n",
       "3          4.5        0.160027         0.785508  \n",
       "4          4.3       -0.858908        -0.142817  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variables = ['Reviews_scaled', 'Category','Price_scaled', 'Types_num', 'Content_num',\n",
    "       'Ratings_imp', 'Size_Mb_scaled', 'Installs_scaled']\n",
    "tmp_df = df[variables]\n",
    "tmp_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dummy variable\n",
    "data = pd.get_dummies(tmp_df, prefix=[\"Category\"], columns=[\"Category\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Reviews_scaled', 'Price_scaled', 'Types_num', 'Content_num',\n",
       "       'Ratings_imp', 'Size_Mb_scaled', 'Installs_scaled',\n",
       "       'Category_ART_AND_DESIGN', 'Category_AUTO_AND_VEHICLES',\n",
       "       'Category_BEAUTY', 'Category_BOOKS_AND_REFERENCE', 'Category_BUSINESS',\n",
       "       'Category_COMICS', 'Category_COMMUNICATION', 'Category_DATING',\n",
       "       'Category_EDUCATION', 'Category_ENTERTAINMENT', 'Category_EVENTS',\n",
       "       'Category_FAMILY', 'Category_FINANCE', 'Category_FOOD_AND_DRINK',\n",
       "       'Category_GAME', 'Category_HEALTH_AND_FITNESS',\n",
       "       'Category_HOUSE_AND_HOME', 'Category_LIBRARIES_AND_DEMO',\n",
       "       'Category_LIFESTYLE', 'Category_MAPS_AND_NAVIGATION',\n",
       "       'Category_MEDICAL', 'Category_NEWS_AND_MAGAZINES', 'Category_PARENTING',\n",
       "       'Category_PERSONALIZATION', 'Category_PHOTOGRAPHY',\n",
       "       'Category_PRODUCTIVITY', 'Category_SHOPPING', 'Category_SOCIAL',\n",
       "       'Category_SPORTS', 'Category_TOOLS', 'Category_TRAVEL_AND_LOCAL',\n",
       "       'Category_VIDEO_PLAYERS', 'Category_WEATHER'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r2 is 0.41154813913173693, mae is 0.31838953740912235 and msr is 0.20635708847411074.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.45426543834426886"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# random forest\n",
    "variables = [\"Content_num\", \"Reviews_scaled\", \"Installs_scaled\", \"Ratings_imp\", 'Category_ART_AND_DESIGN', 'Category_AUTO_AND_VEHICLES',\n",
    "       'Category_BEAUTY', 'Category_BOOKS_AND_REFERENCE', 'Category_BUSINESS',\n",
    "       'Category_COMICS', 'Category_COMMUNICATION', 'Category_DATING',\n",
    "       'Category_EDUCATION', 'Category_ENTERTAINMENT', 'Category_EVENTS',\n",
    "       'Category_FAMILY', 'Category_FINANCE', 'Category_FOOD_AND_DRINK',\n",
    "       'Category_GAME', 'Category_HEALTH_AND_FITNESS',\n",
    "       'Category_HOUSE_AND_HOME', 'Category_LIBRARIES_AND_DEMO',\n",
    "       'Category_LIFESTYLE', 'Category_MAPS_AND_NAVIGATION',\n",
    "       'Category_MEDICAL', 'Category_NEWS_AND_MAGAZINES', 'Category_PARENTING',\n",
    "       'Category_PERSONALIZATION', 'Category_PHOTOGRAPHY',\n",
    "       'Category_PRODUCTIVITY', 'Category_SHOPPING', 'Category_SOCIAL',\n",
    "       'Category_SPORTS', 'Category_TOOLS', 'Category_TRAVEL_AND_LOCAL',\n",
    "       'Category_VIDEO_PLAYERS', 'Category_WEATHER']\n",
    "new_data = data[variables]\n",
    "x = new_data.drop(\"Ratings_imp\", axis=1)\n",
    "y = new_data[\"Ratings_imp\"]\n",
    "x_train, x_test, y_train, y_test = train_test_split(x,y, train_size=0.9, test_size=0.1, random_state=8)\n",
    "\n",
    "rf_model = RandomForestRegressor(max_leaf_nodes=260, random_state=8, n_jobs=-1, n_estimators=80)\n",
    "rf_model.fit(x_train, y_train)\n",
    "rf_pred = rf_model.predict(x_test)\n",
    "r2_rf = r2_score(y_test, rf_pred)\n",
    "msr_rf = mean_squared_error(y_test, rf_pred)\n",
    "mae_rf = mean_absolute_error(y_test, rf_pred)\n",
    "print(f\"r2 is {r2_rf}, mae is {mae_rf} and msr is {msr_rf}.\") \n",
    "math.sqrt(msr_rf)\n",
    "# it seem the model with one hot encoding is a tad better?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Leaf nodes is 10, estimators is 10, r2 is 0.1917770919375088, mae is 0.3824855872948222 and mse is 0.28342594736596854.\n",
      "Leaf nodes is 10, estimators is 20, r2 is 0.20322383920541298, mae is 0.37992973061518925 and mse is 0.27941182557320476.\n",
      "Leaf nodes is 60, estimators is 10, r2 is 0.33850350429674503, mae is 0.345528283513934 and mse is 0.2319722308087154.\n",
      "Leaf nodes is 60, estimators is 20, r2 is 0.33936729363810925, mae is 0.3433076755188741 and mse is 0.23166931893878626.\n",
      "Leaf nodes is 60, estimators is 30, r2 is 0.34379379659425247, mae is 0.3424847720770799 and mse is 0.23011704198480745.\n",
      "Leaf nodes is 60, estimators is 40, r2 is 0.3481138289869925, mae is 0.3419162893213485 and mse is 0.22860210190905628.\n",
      "Leaf nodes is 60, estimators is 50, r2 is 0.34825507951598467, mae is 0.34184111929165417 and mse is 0.22855256846401756.\n",
      "Leaf nodes is 60, estimators is 70, r2 is 0.34919416668054803, mae is 0.3415102103834518 and mse is 0.2282232513082917.\n",
      "Leaf nodes is 60, estimators is 150, r2 is 0.3492462290676762, mae is 0.3413461068258479 and mse is 0.22820499417743492.\n",
      "Leaf nodes is 60, estimators is 180, r2 is 0.3495504769698148, mae is 0.3413306627545173 and mse is 0.2280983011487698.\n",
      "Leaf nodes is 60, estimators is 190, r2 is 0.3502299554192009, mae is 0.3412632706084248 and mse is 0.2278600230434218.\n",
      "Leaf nodes is 60, estimators is 220, r2 is 0.351171216039027, mae is 0.3412248725562343 and mse is 0.2275299437048739.\n",
      "Leaf nodes is 60, estimators is 240, r2 is 0.3520598360660838, mae is 0.34106923771163183 and mse is 0.22721832426114805.\n",
      "Leaf nodes is 110, estimators is 10, r2 is 0.3805196178062459, mae is 0.33077970385566224 and mse is 0.2172381065253369.\n",
      "Leaf nodes is 110, estimators is 30, r2 is 0.383025754020883, mae is 0.3293114764826171 and mse is 0.21635925983121823.\n",
      "Leaf nodes is 110, estimators is 40, r2 is 0.3873381485614211, mae is 0.329282980263105 and mse is 0.21484699818177072.\n",
      "Leaf nodes is 110, estimators is 50, r2 is 0.388363381572601, mae is 0.32916850513930995 and mse is 0.21448747157770415.\n",
      "Leaf nodes is 160, estimators is 20, r2 is 0.38961550270718626, mae is 0.3254906393195024 and mse is 0.2140483803130956.\n",
      "Leaf nodes is 160, estimators is 30, r2 is 0.39692177300864406, mae is 0.3240777200134467 and mse is 0.21148623246842893.\n",
      "Leaf nodes is 160, estimators is 40, r2 is 0.40343172058654697, mae is 0.32370775586432465 and mse is 0.20920333743889674.\n",
      "Leaf nodes is 160, estimators is 50, r2 is 0.4048195977824477, mae is 0.323545990289419 and mse is 0.20871663951787545.\n",
      "Leaf nodes is 160, estimators is 60, r2 is 0.40557734599694173, mae is 0.3228432153531406 and mse is 0.20845091393225365.\n",
      "Leaf nodes is 210, estimators is 40, r2 is 0.41118382730993164, mae is 0.32046591561426446 and mse is 0.20648484459460886.\n",
      "Leaf nodes is 210, estimators is 60, r2 is 0.4133098169983733, mae is 0.32010538613267975 and mse is 0.20573930690256134.\n",
      "Leaf nodes is 210, estimators is 220, r2 is 0.41347056643793534, mae is 0.31950024814286787 and mse is 0.2056829356196616.\n",
      "Leaf nodes is 210, estimators is 230, r2 is 0.4138838619725167, mae is 0.31943438795995305 and mse is 0.20553800199149752.\n",
      "Leaf nodes is 260, estimators is 230, r2 is 0.4140740648976029, mae is 0.31787991608878896 and mse is 0.20547130202086247.\n"
     ]
    }
   ],
   "source": [
    "leaf_nodes = np.arange(10, 1000, 50)\n",
    "estimators = np.arange(10, 500, 10)\n",
    "best_r2 = 0\n",
    "best_mae = 100\n",
    "for nodes in leaf_nodes:\n",
    "    for n in estimators:\n",
    "        tmp_model = RandomForestRegressor(max_leaf_nodes=nodes, random_state=8, n_jobs=-1, n_estimators = n)\n",
    "        tmp_model.fit(x_train, y_train)\n",
    "        tmp_pred = tmp_model.predict(x_test)\n",
    "        tmp_r2 = r2_score(y_test, tmp_pred)\n",
    "        tmp_mae = mean_absolute_error(y_test, tmp_pred)\n",
    "        tmp_mse = mean_squared_error(y_test, tmp_pred)\n",
    "        if (tmp_r2 > best_r2) and (tmp_mae < best_mae):\n",
    "            best_r2 = tmp_r2\n",
    "            best_mae = tmp_mae\n",
    "            print(f\"Leaf nodes is {nodes}, estimators is {n}, r2 is {tmp_r2}, mae is {tmp_mae} and mse is {tmp_mse}.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R2 is 0.4140740648976029, MSE is 0.20547130202086247, MAE is 0.31787991608878896\n"
     ]
    }
   ],
   "source": [
    "final_model = RandomForestRegressor(max_leaf_nodes=260, n_estimators=230, random_state=8, n_jobs=-1)\n",
    "final_model.fit(x_train, y_train)\n",
    "final_pred = final_model.predict(x_test)\n",
    "final_r2 = r2_score(y_test, final_pred)\n",
    "final_mse = mean_squared_error(y_test, final_pred)\n",
    "final_mae = mean_absolute_error(y_test, final_pred)\n",
    "final_r2, final_mse, final_mae  # 0.4140740648976029, 0.2054713020208625, 0.31787991608878896)\n",
    "print(f\"R2 is {final_r2}, MSE is {final_mse}, MAE is {final_mae}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.45328942412200895"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "math.sqrt(final_mse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that the best model I could come up with is using Random Forest Regressor (max_leaf_nodes=260, n_estimators=230, random_state=8, n_jobs=-1) using the following variables:\n",
    "\n",
    "- Content_num: The encoded Content Rating, ie \"Everyone\", \"Teen\", etc.\n",
    "- Reviews_scaled: Number of reviews standardised.\n",
    "- Installs_scaled: Number of installds standardised.\n",
    "- The categories that have been encoded into dummy variables.\n",
    "\n",
    "Using the variables and the variables with standardised data, the model achieved roughly 41% accuracy with 0.21 mean absolute error and root mean-squared error of 0.45. It's not the most ideal but it's the best I can achieve at the moment and while it is likely that there are better models, I think there's a possibility that the variables here can't completely predict the average ratings of an app. For example, if you look at the correlation matrix (from the data analysis), ratings are not strongly correlated with any variables. I understand that the dummy variables are a lot (33 in total!), it gave me overall better, albeit slightly, metrics. Let me know what you think, but be nice!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}