tutorials/reduce_model_complexity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the model-complexity-reducer (mcr) and what is it for?\n",
    "\n",
    "The mcr tool is designed to maximize the tradeoff between the complexity of GBT algorithms and their performance. In this context, by complexity, we refer to the number of operations the algorithm requires to perform an inference (number of trees and their depth).\n",
    "\n",
    "This tool is specifically designed so that the resulting model is manageable within the ZKML paradigm, but its functionality can be extended to any problem where:\n",
    "\n",
    "- The model's weight needs to be minimal, for example for mobile applications.\n",
    "- Minimal inference times are required for low latency applications.\n",
    "- We want to check if we have created an overly complex model and a simpler one would give us the same performance (or even better).\n",
    "- The number of steps required to perform the inference must be less than X (as is currently constrained by the ZKML paradigm).\n",
    "\n",
    "Now that we understand the possible applications of this tool, let's see how it works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How does mcr work?\n",
    "\n",
    "The best way to understand how our dimensionality reduction algorithm works is to explain step by step the operations it performs:\n",
    "\n",
    "- Given a model, it retrieves the type of model it is and its important parameters.\n",
    "- Based on these parameters, it adjusts the search space so that the architectures it will try to find are lighter than the original.\n",
    "- It carries out a Bayesian optimization process, where it adds as possible hyperparameters:\n",
    "    - Model-specific parameters adjusted to the new search space.\n",
    "    - Transformations of the input data using different dimensionality reduction techniques.\n",
    "- After obtaining the best parameters for a given evaluation metric, it returns:\n",
    "    - A transformer with the transformation that must be applied to the test set to generate the inference.\n",
    "    - The newly adjusted model.\n",
    "\n",
    "Within the world of ZKML, we know that the cost of executing a ZK Proof is much higher than the cost of generating a normal inference, due to the cryptographic process involved.\n",
    "In this context, mcr has been developed with the goal of creating models that can be transpiled into others. These new models can be interpreted by other programs (in our case, Cairo) and generate ZK Proofs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For this example, it is necessary to have both xgboost and lightgbm installed, but it is not necessary to have all packages installed to use zkcook. \n",
    "# For this reason, we include this cell to ensure the notebook works correctly.\n",
    "\n",
    "!pip install xgboost\n",
    "!pip install lightgbm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In this simple tutorial, we will only look at how to use the package, without delving into performance details. To see these details, please refer to the notebook end_to_end_using_giza_stack.ipynb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lightgbm as lgbm\n",
    "from sklearn.datasets import load_diabetes\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data = load_diabetes()\n",
    "X, y = data.data, data.target\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "n_estimators = 1200\n",
    "max_depth = 8\n",
    "\n",
    "lgbm_reg = lgbm.LGBMRegressor(n_estimators=n_estimators, max_depth=max_depth)\n",
    "lgbm_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'boosting_type': 'gbdt',\n",
       " 'class_weight': None,\n",
       " 'colsample_bytree': 1.0,\n",
       " 'importance_type': 'split',\n",
       " 'learning_rate': 0.1,\n",
       " 'max_depth': 8,\n",
       " 'min_child_samples': 20,\n",
       " 'min_child_weight': 0.001,\n",
       " 'min_split_gain': 0.0,\n",
       " 'n_estimators': 1200,\n",
       " 'n_jobs': None,\n",
       " 'num_leaves': 31,\n",
       " 'objective': None,\n",
       " 'random_state': None,\n",
       " 'reg_alpha': 0.0,\n",
       " 'reg_lambda': 0.0,\n",
       " 'subsample': 1.0,\n",
       " 'subsample_for_bin': 200000,\n",
       " 'subsample_freq': 0}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgbm_reg.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from giza.zkcook import mcr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model, transformer = mcr(model = lgbm_reg,\n",
    "                         X_train = X_train,\n",
    "                         y_train = y_train, \n",
    "                         X_eval = X_test, \n",
    "                         y_eval = y_test, \n",
    "                         eval_metric = 'rmse', \n",
    "                         transform_features = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'boosting_type': 'gbdt',\n",
       " 'class_weight': None,\n",
       " 'colsample_bytree': 1.0,\n",
       " 'importance_type': 'split',\n",
       " 'learning_rate': 0.1,\n",
       " 'max_depth': 3,\n",
       " 'min_child_samples': 20,\n",
       " 'min_child_weight': 0.001,\n",
       " 'min_split_gain': 0.0,\n",
       " 'n_estimators': 200,\n",
       " 'n_jobs': None,\n",
       " 'num_leaves': 31,\n",
       " 'objective': None,\n",
       " 'random_state': None,\n",
       " 'reg_alpha': 0.0,\n",
       " 'reg_lambda': 0.0,\n",
       " 'subsample': 1.0,\n",
       " 'subsample_for_bin': 200000,\n",
       " 'subsample_freq': 0,\n",
       " 'min_data_in_leaf': 72,\n",
       " 'feature_fraction': 0.16836383405441807,\n",
       " 'bagging_fraction': 0.24326086678781003,\n",
       " 'verbose': -1,\n",
       " 'early_stopping_rounds': 10}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can observe, the complexity of the model has drastically decreased. We have gone from 1200 trees of depth 8 to 200 trees of depth 3. In the notebook end_to_end_example.ipynb, we will see the performance of our new models after reducing their complexity.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have our model and our transformer ready for making inferences!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mcr_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}