usage_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example usage of the CBP_KRP algorithm\n",
    "\n",
    "**Original publication:** López-Sánchez, Daniel, Angélica González Arrieta, and Juan M. Corchado. \"Compact bilinear pooling via kernelized random projection for fine-grained image categorization on low computational power devices.\" Neurocomputing 398 (2020): 411-421."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main motivation of the CBP_KRP method is to achieve fast inference times when running on low computational\n",
    "power devices. In this repo, two implementations of CBP_KRP are provided. The first one is a toch-based implementation designed to run efficiently on GPU devices. Additionally, we provide a scipy/numpy-based implementation, optimized for fast inference on low computational power devices. The workflow poposed in the paper involves training/fine-tuning your models end-to-end using the torch implementation, and they deploying the models\n",
    "to low computational power devices using the scipy/numpy-based implementation of CBP_KRP, which takes advantage of several\n",
    "computational tricks such as the use of sparse matrix multiplication routines.\n",
    "\n",
    "Some of the tricks described in the paper are not really effective when running on powerful desktop\n",
    "CPUs, where parallelization can play a much larger role than the computational tricks of CBP_KRP. To simulate CPU inference on a low computational power device, one can set some environment variables to limit\n",
    "the number of threads used by lineal algebra libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# To simulate CPU inference on a low computational power device, we can set these\n",
    "# env variables to limit the number of threads used by numpy.\n",
    "os.environ['OPENBLAS_NUM_THREADS'] = '1'\n",
    "os.environ['MKL_NUM_THREADS'] = '1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "from CBP_KRP import CBP_KRP, CBP_KRP_cpu_inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an example batch with random data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 32\n",
    "CHANNELS = 1024\n",
    "FEATURE_MAP_HEIGHT = 50\n",
    "FEATURE_MAP_WIDTH = 50\n",
    "\n",
    "example_batch = torch.Tensor(np.random.rand(BATCH_SIZE, CHANNELS, FEATURE_MAP_HEIGHT, FEATURE_MAP_WIDTH))\n",
    "example_batch_numpy = example_batch.detach().numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a CBP_KRP module for GPU inference, and run a forward pass on the example batch. See the original paper for a description of the hyper-parameters of the algorithm. \n",
    "\n",
    "We keep the resulting descriptors for comparison with the cpu inference implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "S = 100 # Sparsity level of the random vectors\n",
    "K = 512 # Number of features of the output descriptor\n",
    "P = 5000 # Total number of unique random vectors used by CBP_KRP\n",
    "T = 2 # Number of vectors to sum for the CLT\n",
    "\n",
    "cbp_krp = CBP_KRP(input_dim=INPUT_DIM, k=K, s=S, p=P, t=T)\n",
    "\n",
    "compact_bilinear_pooling_descriptors_torch = cbp_krp.forward(example_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a CBP_KRP module for CPU inference in low computational power devices. For initialization, we use the random\n",
    "vectors and indexes created by the GPU instance of CBP_KRP, so the results are the same. Note that this CPU implemantation uses several tricks for efficient inference in low computational power devices, such as the use of sparse matrix multiplication routines.\n",
    "\n",
    "Run a forward pass\n",
    "on the example batch and keep the resulting descriptors for comparison with the GPU inference implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "cbp_krp_cpu = CBP_KRP_cpu_inference(random_vectors=cbp_krp.random_vectors.detach().numpy(),\n",
    "                                    random_indexes=cbp_krp.random_indexes,\n",
    "                                    s=cbp_krp.s, use_sparse_matrix_multiplication=True)\n",
    "\n",
    "compact_bilinear_pooling_descriptors_cpu = cbp_krp_cpu.forward(example_batch_numpy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Up to numerical precission errors, the results of the toch and numpy/scipy implementations should be the same:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Some GPU results:  tensor([-56082.5078,  11413.1699,  -9906.0254, -14039.8086])\n",
      "Some CPU results:  [-56082.47680664  11413.16070557  -9906.02264404 -14039.81323242]\n"
     ]
    }
   ],
   "source": [
    "print(\"Some GPU results: \", compact_bilinear_pooling_descriptors_torch[0, :4])\n",
    "print(\"Some CPU results: \", compact_bilinear_pooling_descriptors_cpu[0, :4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " In a device with low computational power (e.g., few cores), the use of sparse matrix multiplication\n",
    " routines can make a big difference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.19 s ± 61.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "cbp_krp_cpu = CBP_KRP_cpu_inference(random_vectors=cbp_krp.random_vectors.detach().numpy(),\n",
    "                                    random_indexes=cbp_krp.random_indexes,\n",
    "                                    s=cbp_krp.s, use_sparse_matrix_multiplication=True)\n",
    "\n",
    "%timeit compact_bilinear_pooling_descriptors_cpu = cbp_krp_cpu.forward(example_batch_numpy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8.28 s ± 130 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "cbp_krp_cpu = CBP_KRP_cpu_inference(random_vectors=cbp_krp.random_vectors.detach().numpy(),\n",
    "                                    random_indexes=cbp_krp.random_indexes,\n",
    "                                    s=cbp_krp.s, use_sparse_matrix_multiplication=False)\n",
    "\n",
    "%timeit compact_bilinear_pooling_descriptors_cpu = cbp_krp_cpu.forward(example_batch_numpy)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}