WhatIsPants.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "63d77145-0343-41e3-a71f-2be7d2d60b70",
   "metadata": {},
   "source": [
    "# What is pants?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8665b48e-aeb7-4050-b6af-e0e27a0fe9c7",
   "metadata": {},
   "source": [
    "I have started this project to understand how segmentation models work. Identifying and segmenting pants in an image is a fairly easy task for humans — we can do it with almost 100% accuracy. But how well can machines do this task? To answer this existential question, I decided to train a segmentation model and run some predictions. My first choice was the Ultralytics YOLOv8 segmentation model, because it's well-documented, open-source, and looks very promising. To read more about the whole project, check my Medium article: \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04edc72e-f4cf-409d-9c4c-e660fe4918a3",
   "metadata": {},
   "source": [
    "## Setting up Google Colab env\n",
    "If you're running this as a Jupyter notebook from an already cloned git repository, feel free to skip this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b3b00a4-8a3e-473a-9e07-f048402835fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# clone the repo\n",
    "!git clone https://github.com/LorenaDerezanin/WhatIsPants.git\n",
    "%cd WhatIsPants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09a391ca-81fa-4a13-8fdc-eaf460bad96b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# install requirements with pip\n",
    "!pip install -r requirements.txt --no-cache-dir"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac570832-6a80-4e0c-9baf-086a545a577d",
   "metadata": {},
   "source": [
    "## Prepare dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a5f416-efa2-4933-b47e-e33f8729672b",
   "metadata": {},
   "source": [
    "As our initial dataset we will use the Deep Fashion MultiModal dataset: https://github.com/yumingj/DeepFashion-MultiModal    \n",
    "    * from 44,096 jpg images, 12,701 are annotated (classes, segmentation masks and bounding boxes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "613d2cd0-2105-4bca-80ce-ec67d40359f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pathlib\n",
    "# Set an environment variable for the project home\n",
    "os.environ[\"PROJECT_HOME\"] = os.path.abspath(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5713b193-5432-4984-b335-1519dd489f3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/lorenaderezanin/PycharmProjects/WhatIsPants'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%env PROJECT_HOME\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75c744c2-4c8a-4733-aef1-157ac5218e82",
   "metadata": {},
   "outputs": [],
   "source": [
    "# download image files\n",
    "!wget --header 'Sec-Fetch-Dest: document' \\\n",
    "  'https://drive.usercontent.google.com/download?id=1U2PljA7NE57jcSSzPs21ZurdIPXdYZtN&export=download&authuser=0&confirm=t&uuid=115a0cd6-8ddb-427b-9343-62b76c4d939c&at=APZUnTWiXg4LlG3A7QPA5DmjASX8%3A1715537567680' \\\n",
    "  --output-document 'images.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc12e094-248c-46ca-a9fd-c0549ceed114",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# download annotation labels\n",
    "!wget --header 'Sec-Fetch-Dest: document' \\\n",
    "  'https://drive.usercontent.google.com/download?id=1r-5t-VgDaAQidZLVgWtguaG7DvMoyUv9&export=download&authuser=0&confirm=t&uuid=b445e6d2-634c-4b59-96c8-4455c6f117a5&at=APZUnTV7OltdPbT0OB1lUK1FhJO8%3A1715537716467' \\\n",
    "  --output-document 'segm.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "982bed30-21d6-4782-821e-0095aaa52b25",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# unzip the downloaded segmentation labels\n",
    "!rm -rf datasets/deepfashion/segm\n",
    "!rm -rf datasets/deepfashion/labels\n",
    "!mkdir -p datasets/deepfashion/labels\n",
    "!unzip -qo segm.zip -d datasets/deepfashion/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8e568ef-4aa8-4ea7-a0df-1648047457a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove data from target directory in preparation for unzipping \n",
    "!rm -rf datasets/deepfashion/images\n",
    "\n",
    "# unzip the downloaded images\n",
    "# this takes about 2 minutes\n",
    "# tqdm is used to show a progress bar\n",
    "!unzip images.zip -d datasets/deepfashion/ | tqdm --desc extracted --unit files --unit_scale --total 44097 > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de97d0c-5e0c-4391-a7b3-9d5d1e84f2f6",
   "metadata": {},
   "source": [
    "### Convert masks to contours format that YOLO can process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10bca7fe-fbbb-4245-9199-7501f70afff6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import concurrent.futures\n",
    "import numpy as np\n",
    "import os\n",
    "from tqdm import tqdm\n",
    "\n",
    "# import the mask2contour function \n",
    "from mask_2_contour import mask2contour\n",
    "\n",
    "masks_dir = \"datasets/deepfashion/segm\"\n",
    "labels_dir = \"datasets/deepfashion/labels\"\n",
    "\n",
    "# define the mask color for pants\n",
    "# pants are marked with a light gray color in mask files\n",
    "pants_mask_color = np.array([211, 211, 211])\n",
    "\n",
    "# Get the number of available CPUs\n",
    "num_cpus = os.cpu_count()\n",
    "\n",
    "mask_files = os.listdir(masks_dir)\n",
    "\n",
    "# load labelled mask pngs\n",
    "# parallelize processing\n",
    "with tqdm(total=len(mask_files)) as pbar:\n",
    "    with concurrent.futures.ThreadPoolExecutor(max_workers=num_cpus) as executor:\n",
    "        futures = {\n",
    "            executor.submit(mask2contour, mask_filename, masks_dir, labels_dir, pants_mask_color): mask_filename\n",
    "            for mask_filename in mask_files\n",
    "        }\n",
    "        results = {}\n",
    "        for future in concurrent.futures.as_completed(futures):\n",
    "                arg = futures[future]\n",
    "                results[arg] = future.result()\n",
    "                pbar.update(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6efa680d-7c7d-478f-ba60-2a9f2091cd44",
   "metadata": {},
   "source": [
    "## Subset data into `train`, `val` and `test` sets\n",
    "\n",
    "Dataset containing all 12,701 labelled images was split into:   \n",
    "    * train 80%   \n",
    "    * val 10%  \n",
    "    * test 10%   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fe47f07-d2ad-4772-a9b9-6c5895fb09ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import importlib\n",
    "import subset_training_data\n",
    "importlib.reload(subset_training_data)\n",
    "from subset_training_data import set_up_target_dirs, copy_files_in_parallel\n",
    "\n",
    "# define working directory and subset size\n",
    "WORKDIR = '.'\n",
    "subset_size = 12701\n",
    "\n",
    "num_train_labels = round(0.8 * subset_size)\n",
    "num_val_labels = round(0.1 * subset_size)\n",
    "\n",
    "basedir = os.path.join(WORKDIR, 'datasets', 'deepfashion')\n",
    "labels_source_dir = os.path.join(basedir, 'labels')\n",
    "images_source_dir = os.path.join(basedir, 'images')\n",
    "\n",
    "# setup directories\n",
    "train_dir, val_dir, test_dir = set_up_target_dirs(basedir)\n",
    "\n",
    "# copy files in parallel\n",
    "copy_files_in_parallel(\n",
    "    labels_source_dir,\n",
    "    images_source_dir,\n",
    "    train_dir,\n",
    "    val_dir,\n",
    "    test_dir,\n",
    "    num_train_labels,\n",
    "    num_val_labels,\n",
    "    subset_size\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "323fe7d5-99fd-4aa5-b3b0-e03098759640",
   "metadata": {},
   "source": [
    "### Inspect annotations by printing 4x4 set of labelled images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5e33f8f-bd99-4531-ac4a-3e596d64870d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import supervision as sv\n",
    "from inspect_annotations import load_and_annotate_images, plot_image_grid\n",
    "\n",
    "# define images, labels and yaml paths, and sample size\n",
    "IMAGES_DIRECTORY_PATH = \"datasets/lvis_pants/images/train2017\"\n",
    "ANNOTATIONS_DIRECTORY_PATH = \"datasets/lvis_pants/labels/train2017\"\n",
    "DATA_YAML_PATH = \"lvis.yaml\"\n",
    "SAMPLE_SIZE = 16\n",
    "\n",
    "# load and annotate images\n",
    "images, image_names = load_and_annotate_images(\n",
    "    images_directory_path=IMAGES_DIRECTORY_PATH,\n",
    "    annotations_directory_path=ANNOTATIONS_DIRECTORY_PATH,\n",
    "    data_yaml_path=DATA_YAML_PATH,\n",
    "    sample_size=SAMPLE_SIZE\n",
    ")\n",
    "\n",
    "# plot images grid\n",
    "plot_image_grid(\n",
    "    images=images,\n",
    "    titles=image_names,\n",
    "    grid_size=(4, 4),\n",
    "    size=(16, 16)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e6653e4-d420-417e-aa1c-3380c1534b8a",
   "metadata": {},
   "source": [
    "## Find and prepare a more diverse dataset\n",
    "\n",
    "After a few training and test runs, it became obvious that our model is overfitting and not generalizing well.   \n",
    "To enrich a very uniform initial dataset, let's supplement it with LVIS (Large Vocabulary Instance Segmentation) dataset: https://www.lvisdataset.org/dataset   \n",
    "to create a more diverse set and prevent overfitting.  \n",
    "* LVIS is based on the COC0 2017 train, val and test image sets (~160k images with ~2M instance annotations, and 1203 categories)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96921bc9-d49b-4b41-b84f-31d54a7297ca",
   "metadata": {},
   "source": [
    "# Download LVIS Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0d52a3d-fa6a-431c-a059-15cb4f5f8cf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!echo $PROJECT_HOME\n",
    "!mkdir -p \"$PROJECT_HOME/datasets/lvis\"\n",
    "%cd \"{os.environ['PROJECT_HOME']}/datasets/lvis\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "472943f1-0ce6-47d5-9092-19c75ed69377",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -P datasets/lvis http://images.cocodataset.org/zips/train2017.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d69e71ff-4df9-4b93-8682-235c6d1be69a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -P datasets/lvis http://images.cocodataset.org/zips/val2017.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b80faed0-3aaf-4109-a57c-7e475c8b39c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -P datasets/lvis https://github.com/ultralytics/yolov5/releases/download/v1.0/lvis-labels-segments.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fa74382-5f44-4dd2-bff8-7f0b14e62dab",
   "metadata": {},
   "source": [
    "### Copy LVIS images into dir to be subsetted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a6b790c-303d-4645-bd2d-0e25e4c45276",
   "metadata": {},
   "outputs": [],
   "source": [
    "!unzip train2017.zip -d . | tqdm --desc extracted --unit files --unit_scale --total 118287 > /dev/null \n",
    "!mv train2017 images\n",
    "!unzip val2017.zip -d . | tqdm --desc extracted --unit files --unit_scale --total 5000 > /dev/null\n",
    "!mv val2017/* images/\n",
    "!rm -r val2017"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0245688b-461a-456d-8722-7e29cd97304c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Unzip labels\n",
    "!unzip lvis-labels-segments.zip -d . | tqdm --desc extracted --unit files --unit_scale --total 119018 > /dev/null\n",
    "!mv lvis/labels/train2017 labels\n",
    "!mv lvis/labels/val2017/* labels/\n",
    "!rm -rf lvis\n",
    "!mkdir -p lvis_pants/labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5425493f-d9ea-4c21-a452-8fd7cf707282",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd {os.environ['PROJECT_HOME']}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77657289-3060-4527-9bfd-60ffa031c336",
   "metadata": {},
   "source": [
    "### Subset only pants labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9388de89-d938-48de-b3e8-f978c243a18d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files have been processed and labels adjusted successfully.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import subprocess\n",
    "from subset_lvis_pants_labels import subset_labels\n",
    "\n",
    "# define source and target dirs for training and validation sets\n",
    "source_directory = \"datasets/lvis/labels\"\n",
    "target_directory = \"datasets/lvis_pants/labels\"\n",
    "\n",
    "subset_labels(source_directory, target_directory)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4ddea7cf-4f2c-4256-bf10-e56a183f6e07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    4646\n"
     ]
    }
   ],
   "source": [
    "# Check number of resulting non-empty label files (i.e. which contain pants)\n",
    "!find \"$PROJECT_HOME/datasets/lvis_pants/labels\" -type f -size +0k | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71b2bd47-bf38-41e9-aebe-b37b75c7e272",
   "metadata": {},
   "source": [
    "### Keep only as many pantsless images as there are pantsful images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f3e63a07-d5f5-49fd-bef2-051de97c909d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib\n",
    "import remove_superfluous_empty_labels\n",
    "importlib.reload(remove_superfluous_empty_labels)\n",
    "\n",
    "# define source and target dirs\n",
    "source_images_directory = \"datasets/lvis/images\"\n",
    "source_labels_directory = \"datasets/lvis_pants/labels\"\n",
    "\n",
    "\n",
    "# randomly select the same number of pantless labels as pantsful, \n",
    "# and remove the rest of the empty labels and corresponding images in the set\n",
    "remove_superfluous_empty_labels.remove_empty_labels(source_labels_directory, source_images_directory)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6ef9401c-af88-4925-9256-bcc06410343c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    9292\n"
     ]
    }
   ],
   "source": [
    "# verify that empty labels have been removed by counting remaining label files\n",
    "!ls \"$PROJECT_HOME/datasets/lvis_pants/labels\"  | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa3c82d4-ad1b-4fa7-8a65-3530c2510a39",
   "metadata": {},
   "source": [
    "### Remove images which have no corresponding label file\n",
    "\n",
    "We observed that the LVIS dataset contains images with pants where pants are not annotated. For example: 000000096670.jpg shows a baseball player, and the labels include a baseball, a home base, a bat, and a belt, but no pants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "10315211-0bdd-42ec-95db-50b0e2edb5f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total deleted images: 0\n"
     ]
    }
   ],
   "source": [
    "# Assuming delete_labelless_images.py is in the same directory or properly installed in Python's path\n",
    "from delete_labelless_images import delete_unlabeled_images\n",
    "\n",
    "# Define the directories\n",
    "images_dir = 'datasets/lvis/images'\n",
    "labels_dir = 'datasets/lvis_pants/labels'\n",
    "\n",
    "# Call the function and get the count of deleted files\n",
    "deleted_files_count = delete_unlabeled_images(images_dir, labels_dir)\n",
    "print(f\"Total deleted images: {deleted_files_count}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a42f3efc-cf0b-4840-8131-1d57679819f5",
   "metadata": {},
   "source": [
    "## Split lvis dataset into train and val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "88464c28-a7ac-4818-9284-e5f5cdd776e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import importlib\n",
    "import subset_training_data\n",
    "importlib.reload(subset_training_data)\n",
    "from subset_training_data import set_up_target_dirs, copy_files_in_parallel\n",
    "\n",
    "# define dirs for training and validation sets\n",
    "train_labels_directory = \"datasets/lvis_pants/labels/train\"\n",
    "train_images_directory = \"datasets/lvis_pants/images/train\"\n",
    "val_labels_directory = \"datasets/lvis_pants/labels/val\"\n",
    "val_images_directory = \"datasets/lvis_pants/images/val\"\n",
    "\n",
    "\n",
    "# define subset size\n",
    "\n",
    "subset_size = 9292\n",
    "\n",
    "num_train_labels = round(0.8 * subset_size)\n",
    "num_val_labels = round(0.1 * subset_size)\n",
    "\n",
    "basedir = os.path.join('datasets')\n",
    "labels_source_dir = os.path.join(basedir, 'lvis_pants/labels')\n",
    "images_source_dir = os.path.join(basedir, 'lvis/images')\n",
    "\n",
    "# setup directories\n",
    "train_dir, val_dir, test_dir = set_up_target_dirs(basedir)\n",
    "\n",
    "# copy files in parallel\n",
    "copy_files_in_parallel(\n",
    "    labels_source_dir,\n",
    "    images_source_dir,\n",
    "    train_dir,\n",
    "    val_dir,\n",
    "    test_dir,\n",
    "    num_train_labels,\n",
    "    num_val_labels,\n",
    "    subset_size\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a72b507f-1ef6-4364-922f-ab7549e918d3",
   "metadata": {},
   "source": [
    "### Prepare `train` configuration yaml file \n",
    "\n",
    "We kept only one class in the config yaml (0: This is pants) and removed other object classes. \n",
    "We are now left with `7434` labels in train set, and `929` labels in val set.\n",
    "Tensorboard set to `true` in yaml to monitor model training and validation performance."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1ca2c4a3-f904-4811-82cc-a98262343805",
   "metadata": {},
   "source": [
    "##### Run the training\n",
    "  \n",
    "    * yolo model sizes: s, m, x    \n",
    "    * number of epochs: 5, 25, 50, 100    \n",
    "    * total number of runs: 9    \n",
    "    \n",
    "We found that the precision and recall reached a plateau both in train and val stages around 50th epoch and remained fairly stable until 100th epoch. Same goes for box, class and segmetation loss. Model size s performed a bit more poorly than the larger models.   \n",
    "Comparing the same metrics between model sizes m and x for the same number of epochs was only marginally higher for the larger model x.    \n",
    "Based on these metrics, we concluded that yolo model size m with 50 epochs is an optimal strategy for this task.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0348fef4-f387-4ab2-a0cd-1738d906d8f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import train function\n",
    "from lvis_yolo_train import train_yolo_model\n",
    "\n",
    "# define parameters\n",
    "EPOCHS = 50\n",
    "SIZE = 'm'\n",
    "YAML = 'lvis_fash.yaml'\n",
    "\n",
    "# train YOLO model\n",
    "train_yolo_model(epochs=EPOCHS, size=SIZE, yaml=YAML)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df4ade81-ce82-4df0-8bce-c39faa8e58f8",
   "metadata": {},
   "source": [
    "### Test the model \n",
    "\n",
    "We used a prepared test set to run inferences with our new model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fab4c03-a73d-4c3a-8e6e-122f34152dc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "yolo segment predict model=best.pt source='dataset/test_images/*'"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "whatispants",
   "language": "python",
   "name": "whatispants"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}