RibSegv2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "https://www.upwork.com/jobs/Google-Colab-Developer-Needed-for-Rib-Segmentation-Algorithm-Implementation_~021871373632866913324/?referrer_url_path=find_work_home\n",
        "\n",
        "create a Google Colab environment for implementing the Rib Segmentation Algorithm (Rib Seg V2). The ideal candidate will have experience in setting up Python environments and working with machine learning models in Colab. You should be familiar with algorithm integration and have a strong understanding of image segmentation techniques. If you're technical, detail-oriented, and passionate about building effective solutions, we want to hear from you! Heres the link to it here: https://github.com/M3DV/RibSeg/tree/ribsegv2\n"
      ],
      "metadata": {
        "id": "-gEFWAtmWR8G"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Access to Datasets:\n",
        "\n",
        "RibSeg V2 Dataset: https://drive.google.com/file/d/1ZZGGrhd0y1fLyOZGo_Y-wlVUP4lkHVgm/view?usp=sharing\n",
        "\n",
        "RibSeg V2 Description Document: https://docs.google.com/spreadsheets/d/1lz9liWPy8yHybKCdO3BCA9K76QH8a54XduiZS_9fK70/edit?usp=sharing\n",
        "\n",
        "RibSeg V2 Annotations as Mesh: https://drive.google.com/file/d/1b_qcg99efU8cF2pXshl2ZFxi4LCQOmpw/view?usp=sharing\n",
        "\n",
        "RibFrac Dataset: Accessible via the MICCAI 2020 RibFrac Challenge. Note: You need to join the challenge or request access to download the dataset.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "2f08kiPue38i"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**MICCAI 2020 RibFrac Challenge:**\n",
        "\n",
        "https://ribfrac.grand-challenge.org/dataset/\n",
        "\n",
        "Rib Fracture Detection and Classification\n",
        "Download\n",
        "\n",
        "The RibFrac dataset consists of Training Set, Validation Set and Test Set.  We provide our dataset on zenodo.org. Due to size limit of zenodo.org, we split the whole RibFrac Training Set into 2 parts (Training Set Part 1 and Training Set Part 2).\n",
        "\n",
        "The challenge releases both images and annotations of Training Set and Validation Set; We recommend all participarts to use our offcial data split. Images of Test Set will be released in a short period (refer to Overview: Important Dates), whose prediction results submitted by participants will be evaluated online.\n",
        "\n",
        "Links of all subsets on zenodo.org:\n",
        "\n",
        "RibFrac Dataset: A Benchmark for Rib Fracture Detection, Segmentation and Classification (Training Set Part 1): 300 chest-abdomen CTs and annotations in NII format (nii.gz).\n",
        "RibFrac Dataset: A Benchmark for Rib Fracture Detection, Segmentation and Classification (Training Set Part 2): 120 chest-abdomen CTs and annotations in NII format (nii.gz).\n",
        "RibFrac Dataset: A Benchmark for Rib Fracture Detection, Segmentation and Classification (Tuning/Validation Set): 80 chest-abdomen CTs and annotations in NII format (nii.gz).\n",
        "RibFrac Dataset: A Benchmark for Rib Fracture Detection, Segmentation and Classification (Test Set): 160 chest-abdomen CTs in NII format (nii.gz) without annotations.\n",
        "Ranking will be based on evaluation performed on RibFrac Test Set.\n",
        "\n",
        "Description for info files (ribfrac-train-info-1.csv, ribfrac-train-info-2.csv, ribfrac-val-info.csv)\n",
        "\n",
        "public_id: anonymous patient ID to match images and annotations.\n",
        "label_id: discrete label value in the NII annotations.\n",
        "label_code: 0, 1, 2, 3, 4, -1\n",
        "- 0: it is background\n",
        "- 1: it is a displaced rib fracture\n",
        "- 2: it is a non-displaced rib fracture\n",
        "- 3: it is a buckle rib fracture\n",
        "- 4: it is a segmental rib fracture\n",
        "- -1: it is a rib fracture,  but we could not define its type due to\n",
        "  ambiguity, diagnosis difficulty, etc. Ignore it in the\n",
        "  classification task.\n",
        "For participants from China mainland, we provide a mirror link of CT images. Note that you still need to download annotations from zenodo.org.\n",
        "\n",
        "RibFrac Training Set images (including Part1 and Part2). (提取码: 1wxl)\n",
        "RibFrac Validation Set images.  (提取码: whs1)\n",
        "RibFrac Test Set images. (提取码: rvr8)\n",
        "License\n",
        "The RibFrac dataset is a research effort of thousands of hours by experienced radiologists, computer scientists and engineers. We kindly ask you to respect our effort by appropriate citation and keeping data license.\n",
        "\n",
        "Please note that this dataset is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which means:\n",
        "\n",
        "Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use\n",
        "NonCommercial — You may not use the material for commercial purposes.\n",
        "\n",
        "This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License."
      ],
      "metadata": {
        "id": "yb6DF6oPfS0H"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Clong RibSeg Repo**\n",
        "\n",
        "clone the RibSeg repository and checkout the ribsegv2 branch.\n",
        "\n"
      ],
      "metadata": {
        "id": "lM-N_dWEW1l7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install Git if not already installed\n",
        "!apt-get install git -y\n",
        "\n",
        "# Clone the RibSeg repository\n",
        "!git clone https://github.com/M3DV/RibSeg.git\n",
        "\n",
        "# Navigate to the RibSeg directory\n",
        "%cd RibSeg\n",
        "\n",
        "# Fetch all branches\n",
        "!git fetch --all\n",
        "\n",
        "# Checkout the ribsegv2 branch\n",
        "!git checkout ribsegv2\n",
        "\n",
        "# Verify the current branch\n",
        "!git branch\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K3BskpyeWcLK",
        "outputId": "c72ede4e-4bec-400b-fdb2-fc383013f26d"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "git is already the newest version (1:2.34.1-1ubuntu1.11).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.\n",
            "Cloning into 'RibSeg'...\n",
            "remote: Enumerating objects: 182, done.\u001b[K\n",
            "remote: Counting objects: 100% (29/29), done.\u001b[K\n",
            "remote: Compressing objects: 100% (23/23), done.\u001b[K\n",
            "remote: Total 182 (delta 14), reused 10 (delta 6), pack-reused 153 (from 1)\u001b[K\n",
            "Receiving objects: 100% (182/182), 18.86 MiB | 18.53 MiB/s, done.\n",
            "Resolving deltas: 100% (84/84), done.\n",
            "/content/RibSeg\n",
            "Fetching origin\n",
            "Branch 'ribsegv2' set up to track remote branch 'ribsegv2' from 'origin'.\n",
            "Switched to a new branch 'ribsegv2'\n",
            "  main\u001b[m\n",
            "* \u001b[32mribsegv2\u001b[m\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Copying Necessary Scripts from Main Branch to ribsegv2\n",
        "\n",
        "The ribsegv2 branch may not contain all the necessary scripts required for data preparation, training, and inference. We'll copy these scripts from the main branch into ribsegv2.\n",
        "\n"
      ],
      "metadata": {
        "id": "Ev9E_JyNqQw3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Ensure you're on the ribsegv2 branch\n",
        "!git checkout ribsegv2\n",
        "\n",
        "# List of necessary scripts\n",
        "scripts = ['data_prepare.py', 'inference.py', 'post_proc.py', 'train_ribseg.py', 'test_ribseg.py']\n",
        "\n",
        "# Copy each script from main branch to ribsegv2 branch\n",
        "for script in scripts:\n",
        "    !git checkout main -- {script}\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WAy7tGIUqTPs",
        "outputId": "16558b7b-e898-47c2-f2bb-740708e5cb5f"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Already on 'ribsegv2'\n",
            "Your branch is up to date with 'origin/ribsegv2'.\n",
            "error: pathspec 'test_ribseg.py' did not match any file(s) known to git\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Installing Dependencies\n",
        "\n",
        "RibSeg V2 requires several Python packages. Install them using pip. It's recommended to use a requirements.txt if available, but based on the GitHub brief, we'll install the necessary packages manually."
      ],
      "metadata": {
        "id": "lL53L0BTXVZ-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Upgrade pip\n",
        "!pip install --upgrade pip\n",
        "\n",
        "# Install core dependencies\n",
        "!pip install numpy nibabel torch torchvision torchaudio\n",
        "\n",
        "# Install additional dependencies\n",
        "!pip install scikit-learn matplotlib trimesh open3d pyntcloud h5py tqdm\n",
        "\n",
        "# Install gdown for downloading files from Google Drive\n",
        "!pip install gdown\n",
        "\n",
        "# Install TensorBoard for monitoring training\n",
        "!pip install tensorboard\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E27i_-PiXWiY",
        "outputId": "6845d5d2-f576-4597-bf3e-a8fcf2927d20"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: pip in /usr/local/lib/python3.10/dist-packages (24.1.2)\n",
            "Collecting pip\n",
            "  Downloading pip-24.3.1-py3-none-any.whl.metadata (3.7 kB)\n",
            "Downloading pip-24.3.1-py3-none-any.whl (1.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m30.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: pip\n",
            "  Attempting uninstall: pip\n",
            "    Found existing installation: pip 24.1.2\n",
            "    Uninstalling pip-24.1.2:\n",
            "      Successfully uninstalled pip-24.1.2\n",
            "Successfully installed pip-24.3.1\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (1.26.4)\n",
            "Requirement already satisfied: nibabel in /usr/local/lib/python3.10/dist-packages (5.3.2)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.5.1+cu121)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (0.20.1+cu121)\n",
            "Requirement already satisfied: torchaudio in /usr/local/lib/python3.10/dist-packages (2.5.1+cu121)\n",
            "Requirement already satisfied: importlib-resources>=5.12 in /usr/local/lib/python3.10/dist-packages (from nibabel) (6.4.5)\n",
            "Requirement already satisfied: packaging>=20 in /usr/local/lib/python3.10/dist-packages (from nibabel) (24.2)\n",
            "Requirement already satisfied: typing-extensions>=4.6 in /usr/local/lib/python3.10/dist-packages (from nibabel) (4.12.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.16.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2024.10.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch) (1.3.0)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision) (11.0.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (3.0.2)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (1.6.0)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (3.8.0)\n",
            "Collecting trimesh\n",
            "  Downloading trimesh-4.5.3-py3-none-any.whl.metadata (18 kB)\n",
            "Collecting open3d\n",
            "  Downloading open3d-0.18.0-cp310-cp310-manylinux_2_27_x86_64.whl.metadata (4.2 kB)\n",
            "Collecting pyntcloud\n",
            "  Downloading pyntcloud-0.3.1-py2.py3-none-any.whl.metadata (4.6 kB)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.10/dist-packages (3.12.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (4.67.1)\n",
            "Requirement already satisfied: numpy>=1.19.5 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.26.4)\n",
            "Requirement already satisfied: scipy>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.13.1)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.4.2)\n",
            "Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (3.5.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (4.55.3)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.4.7)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (24.2)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (11.0.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (3.2.0)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (2.8.2)\n",
            "Collecting dash>=2.6.0 (from open3d)\n",
            "  Downloading dash-2.18.2-py3-none-any.whl.metadata (10 kB)\n",
            "Requirement already satisfied: werkzeug>=2.2.3 in /usr/local/lib/python3.10/dist-packages (from open3d) (3.1.3)\n",
            "Requirement already satisfied: nbformat>=5.7.0 in /usr/local/lib/python3.10/dist-packages (from open3d) (5.10.4)\n",
            "Collecting configargparse (from open3d)\n",
            "  Downloading ConfigArgParse-1.7-py3-none-any.whl.metadata (23 kB)\n",
            "Collecting ipywidgets>=8.0.4 (from open3d)\n",
            "  Downloading ipywidgets-8.1.5-py3-none-any.whl.metadata (2.3 kB)\n",
            "Collecting addict (from open3d)\n",
            "  Downloading addict-2.4.0-py3-none-any.whl.metadata (1.0 kB)\n",
            "Requirement already satisfied: pandas>=1.0 in /usr/local/lib/python3.10/dist-packages (from open3d) (2.2.2)\n",
            "Requirement already satisfied: pyyaml>=5.4.1 in /usr/local/lib/python3.10/dist-packages (from open3d) (6.0.2)\n",
            "Collecting pyquaternion (from open3d)\n",
            "  Downloading pyquaternion-0.9.9-py3-none-any.whl.metadata (1.4 kB)\n",
            "Collecting Flask<3.1,>=1.0.4 (from dash>=2.6.0->open3d)\n",
            "  Downloading flask-3.0.3-py3-none-any.whl.metadata (3.2 kB)\n",
            "Collecting werkzeug>=2.2.3 (from open3d)\n",
            "  Downloading werkzeug-3.0.6-py3-none-any.whl.metadata (3.7 kB)\n",
            "Requirement already satisfied: plotly>=5.0.0 in /usr/local/lib/python3.10/dist-packages (from dash>=2.6.0->open3d) (5.24.1)\n",
            "Collecting dash-html-components==2.0.0 (from dash>=2.6.0->open3d)\n",
            "  Downloading dash_html_components-2.0.0-py3-none-any.whl.metadata (3.8 kB)\n",
            "Collecting dash-core-components==2.0.0 (from dash>=2.6.0->open3d)\n",
            "  Downloading dash_core_components-2.0.0-py3-none-any.whl.metadata (2.9 kB)\n",
            "Collecting dash-table==5.0.0 (from dash>=2.6.0->open3d)\n",
            "  Downloading dash_table-5.0.0-py3-none-any.whl.metadata (2.4 kB)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from dash>=2.6.0->open3d) (8.5.0)\n",
            "Requirement already satisfied: typing-extensions>=4.1.1 in /usr/local/lib/python3.10/dist-packages (from dash>=2.6.0->open3d) (4.12.2)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from dash>=2.6.0->open3d) (2.32.3)\n",
            "Collecting retrying (from dash>=2.6.0->open3d)\n",
            "  Downloading retrying-1.3.4-py3-none-any.whl.metadata (6.9 kB)\n",
            "Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from dash>=2.6.0->open3d) (1.6.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from dash>=2.6.0->open3d) (75.1.0)\n",
            "Collecting comm>=0.1.3 (from ipywidgets>=8.0.4->open3d)\n",
            "  Downloading comm-0.2.2-py3-none-any.whl.metadata (3.7 kB)\n",
            "Requirement already satisfied: ipython>=6.1.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets>=8.0.4->open3d) (7.34.0)\n",
            "Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.10/dist-packages (from ipywidgets>=8.0.4->open3d) (5.7.1)\n",
            "Collecting widgetsnbextension~=4.0.12 (from ipywidgets>=8.0.4->open3d)\n",
            "  Downloading widgetsnbextension-4.0.13-py3-none-any.whl.metadata (1.6 kB)\n",
            "Requirement already satisfied: jupyterlab-widgets~=3.0.12 in /usr/local/lib/python3.10/dist-packages (from ipywidgets>=8.0.4->open3d) (3.0.13)\n",
            "Requirement already satisfied: fastjsonschema>=2.15 in /usr/local/lib/python3.10/dist-packages (from nbformat>=5.7.0->open3d) (2.21.1)\n",
            "Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.10/dist-packages (from nbformat>=5.7.0->open3d) (4.23.0)\n",
            "Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /usr/local/lib/python3.10/dist-packages (from nbformat>=5.7.0->open3d) (5.7.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0->open3d) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0->open3d) (2024.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib) (1.17.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=2.2.3->open3d) (3.0.2)\n",
            "Requirement already satisfied: Jinja2>=3.1.2 in /usr/local/lib/python3.10/dist-packages (from Flask<3.1,>=1.0.4->dash>=2.6.0->open3d) (3.1.4)\n",
            "Requirement already satisfied: itsdangerous>=2.1.2 in /usr/local/lib/python3.10/dist-packages (from Flask<3.1,>=1.0.4->dash>=2.6.0->open3d) (2.2.0)\n",
            "Requirement already satisfied: click>=8.1.3 in /usr/local/lib/python3.10/dist-packages (from Flask<3.1,>=1.0.4->dash>=2.6.0->open3d) (8.1.7)\n",
            "Requirement already satisfied: blinker>=1.6.2 in /usr/local/lib/python3.10/dist-packages (from Flask<3.1,>=1.0.4->dash>=2.6.0->open3d) (1.9.0)\n",
            "Collecting jedi>=0.16 (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d)\n",
            "  Downloading jedi-0.19.2-py2.py3-none-any.whl.metadata (22 kB)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (4.4.2)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.7.5)\n",
            "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (3.0.48)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (2.18.0)\n",
            "Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.2.0)\n",
            "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.1.7)\n",
            "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (4.9.0)\n",
            "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.7.0->open3d) (24.3.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.7.0->open3d) (2024.10.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.7.0->open3d) (0.35.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.7.0->open3d) (0.22.3)\n",
            "Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core!=5.0.*,>=4.12->nbformat>=5.7.0->open3d) (4.3.6)\n",
            "Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from plotly>=5.0.0->dash>=2.6.0->open3d) (9.0.0)\n",
            "Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->dash>=2.6.0->open3d) (3.21.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->dash>=2.6.0->open3d) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->dash>=2.6.0->open3d) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->dash>=2.6.0->open3d) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->dash>=2.6.0->open3d) (2024.12.14)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.4 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.8.4)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.7.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.2.13)\n",
            "Downloading trimesh-4.5.3-py3-none-any.whl (704 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m704.8/704.8 kB\u001b[0m \u001b[31m23.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading open3d-0.18.0-cp310-cp310-manylinux_2_27_x86_64.whl (399.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m399.7/399.7 MB\u001b[0m \u001b[31m58.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pyntcloud-0.3.1-py2.py3-none-any.whl (346 kB)\n",
            "Downloading dash-2.18.2-py3-none-any.whl (7.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m154.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading dash_core_components-2.0.0-py3-none-any.whl (3.8 kB)\n",
            "Downloading dash_html_components-2.0.0-py3-none-any.whl (4.1 kB)\n",
            "Downloading dash_table-5.0.0-py3-none-any.whl (3.9 kB)\n",
            "Downloading ipywidgets-8.1.5-py3-none-any.whl (139 kB)\n",
            "Downloading werkzeug-3.0.6-py3-none-any.whl (227 kB)\n",
            "Downloading addict-2.4.0-py3-none-any.whl (3.8 kB)\n",
            "Downloading ConfigArgParse-1.7-py3-none-any.whl (25 kB)\n",
            "Downloading pyquaternion-0.9.9-py3-none-any.whl (14 kB)\n",
            "Downloading comm-0.2.2-py3-none-any.whl (7.2 kB)\n",
            "Downloading flask-3.0.3-py3-none-any.whl (101 kB)\n",
            "Downloading widgetsnbextension-4.0.13-py3-none-any.whl (2.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m93.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading retrying-1.3.4-py3-none-any.whl (11 kB)\n",
            "Downloading jedi-0.19.2-py2.py3-none-any.whl (1.6 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m79.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: dash-table, dash-html-components, dash-core-components, addict, widgetsnbextension, werkzeug, trimesh, retrying, pyquaternion, jedi, configargparse, comm, Flask, pyntcloud, ipywidgets, dash, open3d\n",
            "  Attempting uninstall: widgetsnbextension\n",
            "    Found existing installation: widgetsnbextension 3.6.10\n",
            "    Uninstalling widgetsnbextension-3.6.10:\n",
            "      Successfully uninstalled widgetsnbextension-3.6.10\n",
            "  Attempting uninstall: werkzeug\n",
            "    Found existing installation: Werkzeug 3.1.3\n",
            "    Uninstalling Werkzeug-3.1.3:\n",
            "      Successfully uninstalled Werkzeug-3.1.3\n",
            "  Attempting uninstall: Flask\n",
            "    Found existing installation: Flask 3.1.0\n",
            "    Uninstalling Flask-3.1.0:\n",
            "      Successfully uninstalled Flask-3.1.0\n",
            "  Attempting uninstall: ipywidgets\n",
            "    Found existing installation: ipywidgets 7.7.1\n",
            "    Uninstalling ipywidgets-7.7.1:\n",
            "      Successfully uninstalled ipywidgets-7.7.1\n",
            "Successfully installed Flask-3.0.3 addict-2.4.0 comm-0.2.2 configargparse-1.7 dash-2.18.2 dash-core-components-2.0.0 dash-html-components-2.0.0 dash-table-5.0.0 ipywidgets-8.1.5 jedi-0.19.2 open3d-0.18.0 pyntcloud-0.3.1 pyquaternion-0.9.9 retrying-1.3.4 trimesh-4.5.3 werkzeug-3.0.6 widgetsnbextension-4.0.13\n",
            "Requirement already satisfied: gdown in /usr/local/lib/python3.10/dist-packages (5.2.0)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from gdown) (4.12.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from gdown) (3.16.1)\n",
            "Requirement already satisfied: requests[socks] in /usr/local/lib/python3.10/dist-packages (from gdown) (2.32.3)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from gdown) (4.67.1)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->gdown) (2.6)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (2024.12.14)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (1.7.1)\n",
            "Requirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (2.17.1)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (1.68.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (3.7)\n",
            "Requirement already satisfied: numpy>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (1.26.4)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorboard) (24.2)\n",
            "Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (4.25.5)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (75.1.0)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (1.17.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (3.0.6)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard) (3.0.2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Mount Google Drive**\n",
        "\n",
        "If your datasets are large or you prefer to store them in Google Drive, you can mount your drive to Colab."
      ],
      "metadata": {
        "id": "mVluaI8PXdRE"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "335CFTsBWRNc",
        "outputId": "7545e18e-ee84-450b-cf4d-a1be7f386715"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Downloading the RibSeg V2 Dataset**\n",
        "\n",
        "Download the RibSeg V2 dataset from the provided Google Drive link and place it in the appropriate directory."
      ],
      "metadata": {
        "id": "31J2L6uFYBob"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import zipfile\n",
        "import gdown\n",
        "\n",
        "# Define the URL and output path for the RibSeg V2 dataset\n",
        "ribsegv2_dataset_url = 'https://drive.google.com/uc?id=1ZZGGrhd0y1fLyOZGo_Y-wlVUP4lkHVgm'\n",
        "ribsegv2_zip_path = '/content/RibSeg/data/ribsegv2.zip'\n",
        "\n",
        "# Create the data directory if it doesn't exist\n",
        "os.makedirs('/content/RibSeg/data', exist_ok=True)\n",
        "\n",
        "# Download the dataset using gdown\n",
        "!gdown --id 1ZZGGrhd0y1fLyOZGo_Y-wlVUP4lkHVgm -O {ribsegv2_zip_path}\n",
        "\n",
        "# Extract the dataset\n",
        "with zipfile.ZipFile(ribsegv2_zip_path, 'r') as zip_ref:\n",
        "    zip_ref.extractall('/content/RibSeg/data/ribsegv2')\n",
        "\n",
        "# Remove the zip file to save space\n",
        "os.remove(ribsegv2_zip_path)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uYwb8_mUZiOM",
        "outputId": "a3ac32b7-125b-4b7c-d7b3-30809236c2f8"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/gdown/__main__.py:140: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.\n",
            "  warnings.warn(\n",
            "Downloading...\n",
            "From (original): https://drive.google.com/uc?id=1ZZGGrhd0y1fLyOZGo_Y-wlVUP4lkHVgm\n",
            "From (redirected): https://drive.google.com/uc?id=1ZZGGrhd0y1fLyOZGo_Y-wlVUP4lkHVgm&confirm=t&uuid=eca1b207-48e4-4e84-be35-831d6ded2fed\n",
            "To: /content/RibSeg/data/ribsegv2.zip\n",
            "100% 608M/608M [00:11<00:00, 50.8MB/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Downloading the RibSeg V2 Description Document**\n",
        "\n",
        "Access the RibSeg V2 description document from Google Sheets. Since it's a spreadsheet, it's best to keep it in Google Drive for reference."
      ],
      "metadata": {
        "id": "dloKNgD4ZxfD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install gdown to download files from Google Drive\n",
        "!pip install gdown\n",
        "import os\n",
        "\n",
        "# Define the URL and output path for the Google Sheet\n",
        "google_sheet_url = 'https://docs.google.com/spreadsheets/d/1lz9liWPy8yHybKCdO3BCA9K76QH8a54XduiZS_9fK70/export?format=xlsx'\n",
        "output_path = '/content/RibSeg/data/ribsegv2_description.xlsx'\n",
        "\n",
        "# Download the Google Sheet and save as an Excel file\n",
        "!wget -O $output_path $google_sheet_url\n",
        "\n",
        "# Confirm the download\n",
        "if not os.path.exists(output_path):\n",
        "    print(\"Download failed.\")\n",
        "else:\n",
        "    print(f\"Downloaded file is saved to: {output_path}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tUUfmwtcYFTr",
        "outputId": "cbb0e362-ef69-4573-9fdf-6cb98acbc0eb"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: gdown in /usr/local/lib/python3.10/dist-packages (5.2.0)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from gdown) (4.12.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from gdown) (3.16.1)\n",
            "Requirement already satisfied: requests[socks] in /usr/local/lib/python3.10/dist-packages (from gdown) (2.32.3)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from gdown) (4.67.1)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->gdown) (2.6)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (2024.12.14)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (1.7.1)\n",
            "--2024-12-24 08:01:49--  https://docs.google.com/spreadsheets/d/1lz9liWPy8yHybKCdO3BCA9K76QH8a54XduiZS_9fK70/export?format=xlsx\n",
            "Resolving docs.google.com (docs.google.com)... 142.251.2.102, 142.251.2.113, 142.251.2.101, ...\n",
            "Connecting to docs.google.com (docs.google.com)|142.251.2.102|:443... connected.\n",
            "HTTP request sent, awaiting response... 307 Temporary Redirect\n",
            "Location: https://doc-0g-b0-sheets.googleusercontent.com/export/54bogvaave6cua4cdnls17ksc4/j3028mtvvjajgkk9rq96360ltk/1735027310000/110401750412225248239/*/1lz9liWPy8yHybKCdO3BCA9K76QH8a54XduiZS_9fK70?format=xlsx [following]\n",
            "Warning: wildcards not supported in HTTP.\n",
            "--2024-12-24 08:01:50--  https://doc-0g-b0-sheets.googleusercontent.com/export/54bogvaave6cua4cdnls17ksc4/j3028mtvvjajgkk9rq96360ltk/1735027310000/110401750412225248239/*/1lz9liWPy8yHybKCdO3BCA9K76QH8a54XduiZS_9fK70?format=xlsx\n",
            "Resolving doc-0g-b0-sheets.googleusercontent.com (doc-0g-b0-sheets.googleusercontent.com)... 142.251.2.132, 2607:f8b0:4023:c0d::84\n",
            "Connecting to doc-0g-b0-sheets.googleusercontent.com (doc-0g-b0-sheets.googleusercontent.com)|142.251.2.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: unspecified [application/vnd.openxmlformats-officedocument.spreadsheetml.sheet]\n",
            "Saving to: ‘/content/RibSeg/data/ribsegv2_description.xlsx’\n",
            "\n",
            "/content/RibSeg/dat     [ <=>                ]  17.26K  --.-KB/s    in 0.004s  \n",
            "\n",
            "2024-12-24 08:01:50 (4.70 MB/s) - ‘/content/RibSeg/data/ribsegv2_description.xlsx’ saved [17671]\n",
            "\n",
            "Downloaded file is saved to: /content/RibSeg/data/ribsegv2_description.xlsx\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Downloading the RibSeg V2 Annotations as Mesh**\n",
        "\n",
        "Download the RibSeg V2 annotations provided as mesh files."
      ],
      "metadata": {
        "id": "Ge5pwvJWZ9Tm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the URL and output path for the RibSeg V2 annotations as mesh\n",
        "annotations_mesh_url = 'https://drive.google.com/uc?id=1b_qcg99efU8cF2pXshl2ZFxi4LCQOmpw'\n",
        "annotations_zip_path = '/content/RibSeg/data/ribsegv2_annotations_mesh.zip'\n",
        "\n",
        "# Download the annotations using gdown\n",
        "!gdown --id 1b_qcg99efU8cF2pXshl2ZFxi4LCQOmpw -O {annotations_zip_path}\n",
        "\n",
        "# Create the destination directory\n",
        "os.makedirs('/content/RibSeg/data/ribsegv2_annotations_mesh', exist_ok=True)\n",
        "\n",
        "# Extract the annotations\n",
        "with zipfile.ZipFile(annotations_zip_path, 'r') as zip_ref:\n",
        "    zip_ref.extractall('/content/RibSeg/data/ribsegv2_annotations_mesh')\n",
        "\n",
        "# Remove the zip file to save space\n",
        "os.remove(annotations_zip_path)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vYIN79WVZ-dD",
        "outputId": "d0580a02-eeea-4754-9ad7-cfb7655553da"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/gdown/__main__.py:140: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.\n",
            "  warnings.warn(\n",
            "Downloading...\n",
            "From (original): https://drive.google.com/uc?id=1b_qcg99efU8cF2pXshl2ZFxi4LCQOmpw\n",
            "From (redirected): https://drive.google.com/uc?id=1b_qcg99efU8cF2pXshl2ZFxi4LCQOmpw&confirm=t&uuid=6c82bcc5-247c-4b6f-952f-9ca0d6f1ed6d\n",
            "To: /content/RibSeg/data/ribsegv2_annotations_mesh.zip\n",
            "100% 5.64G/5.64G [01:15<00:00, 74.6MB/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Make a directory in MyDrive**\n",
        "\n",
        "/content/drive/MyDrive/upwork/RibSeg/RibFrac\n",
        "\n"
      ],
      "metadata": {
        "id": "87m4v8Z-sLLK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "# Mount Google Drive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# Define the directory path\n",
        "directory_path = '/content/drive/MyDrive/upwork/RibSeg/RibFrac'\n",
        "\n",
        "# Create the directory if it doesn't exist\n",
        "os.makedirs(directory_path, exist_ok=True)\n",
        "\n",
        "# Confirm the directory was created\n",
        "if os.path.exists(directory_path):\n",
        "    print(f\"Directory created successfully at: {directory_path}\")\n",
        "else:\n",
        "    print(\"Failed to create directory.\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KD6smVEdsZ3J",
        "outputId": "9d3e0846-c1cb-43e4-b57c-f64a068d91b2"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n",
            "Directory created successfully at: /content/drive/MyDrive/upwork/RibSeg/RibFrac\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Downloading the RibFrac Dataset**\n",
        "\n",
        "The RibFrac Dataset is essential for training the RibSeg V2 model.\n",
        "\n",
        "It consists of multiple parts: Training Set Part 1, Training Set Part 2, Validation Set, and Test Set.\n",
        "\n",
        "\n",
        "Steps:\n",
        "\n",
        "Visit the RibFrac Challenge Page.\n",
        "Click on Join Challenge and follow the instructions to gain access.\n",
        "\n",
        "Download the following subsets:\n",
        "\n",
        "*   Training Set Part 1: https://doi.org/10.5281/zenodo.3893507 [300 chest-abdomen CTs and annotations in NII format (nii.gz).]\n",
        "Link: https://zenodo.org/records/3893508/files/ribfrac-train-images-1.zip?download=1\n",
        "\n",
        "*   Training Set Part 2: https://doi.org/10.5281/zenodo.3893497 [120 chest-abdomen CTs and annotations in NII format (nii.gz).]\n",
        "Link: https://zenodo.org/records/3893498/files/ribfrac-train-images-2.zip?download=1\n",
        "\n",
        "*   Validation Set: https://doi.org/10.5281/zenodo.3893495 [80 chest-abdomen CTs and annotations in NII format (nii.gz).]\n",
        "Link: https://zenodo.org/records/3893496/files/ribfrac-val-images.zip?download=1\n",
        "\n",
        "*   Test Set: https://zenodo.org/record/3993380 [160 chest-abdomen CTs in NII format (nii.gz) without annotations.]\n",
        "Link: https://zenodo.org/records/3993380/files/ribfrac-test-images.zip?download=1\n"
      ],
      "metadata": {
        "id": "xfBJ3y63g_I6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import zipfile\n",
        "import gdown\n",
        "\n",
        "# Define the Google Drive path where the RibFrac dataset will be saved\n",
        "ribfrac_drive_path = '/content/drive/MyDrive/upwork/RibSeg/RibFrac'\n",
        "\n",
        "# Create the RibFrac directory in Google Drive if it doesn't exist\n",
        "os.makedirs(ribfrac_drive_path, exist_ok=True)\n",
        "\n",
        "# List of RibFrac dataset parts with their corresponding download links and filenames\n",
        "ribfrac_datasets = {\n",
        "    'ribfrac-train-images-1.zip': 'https://zenodo.org/records/3893508/files/ribfrac-train-images-1.zip?download=1',\n",
        "    'ribfrac-train-images-2.zip': 'https://zenodo.org/records/3893498/files/ribfrac-train-images-2.zip?download=1',\n",
        "    'ribfrac-val-images.zip': 'https://zenodo.org/records/3893496/files/ribfrac-val-images.zip?download=1',\n",
        "    'ribfrac-test-images.zip': 'https://zenodo.org/records/3993380/files/ribfrac-test-images.zip?download=1'\n",
        "}\n",
        "\n",
        "# Function to download files using wget\n",
        "def download_ribfrac(file_name, url, destination_folder):\n",
        "    output_path = os.path.join(destination_folder, file_name)\n",
        "    if not os.path.exists(output_path):\n",
        "        print(f'Downloading {file_name}...')\n",
        "        !wget -c \"{url}\" -O \"{output_path}\"\n",
        "    else:\n",
        "        print(f'{file_name} already exists. Skipping download.')\n",
        "\n",
        "# Download all RibFrac dataset parts\n",
        "for file_name, url in ribfrac_datasets.items():\n",
        "    download_ribfrac(file_name, url, ribfrac_drive_path)\n",
        "\n",
        "# Define extraction paths\n",
        "extraction_paths = {\n",
        "    'ribfrac-train-images-1.zip': '/content/RibSeg/data/ribfrac/train_part1',\n",
        "    'ribfrac-train-images-2.zip': '/content/RibSeg/data/ribfrac/train_part2',\n",
        "    'ribfrac-val-images.zip': '/content/RibSeg/data/ribfrac/val',\n",
        "    'ribfrac-test-images.zip': '/content/RibSeg/data/ribfrac/test'\n",
        "}\n",
        "\n",
        "# Extract the datasets\n",
        "def extract_zip(file_name, extract_to):\n",
        "    zip_path = os.path.join(ribfrac_drive_path, file_name)\n",
        "    if not os.path.exists(extract_to):\n",
        "        os.makedirs(extract_to, exist_ok=True)\n",
        "        print(f'Extracting {file_name}...')\n",
        "        with zipfile.ZipFile(zip_path, 'r') as zip_ref:\n",
        "            zip_ref.extractall(extract_to)\n",
        "    else:\n",
        "        print(f'{extract_to} already exists. Skipping extraction.')\n",
        "\n",
        "for file_name, extract_to in extraction_paths.items():\n",
        "    extract_zip(file_name, extract_to)\n",
        "\n",
        "# Optionally, remove the zip files from Google Drive to save space\n",
        "for file_name in ribfrac_datasets.keys():\n",
        "    zip_path = os.path.join(ribfrac_drive_path, file_name)\n",
        "    if os.path.exists(zip_path):\n",
        "        print(f'Removing {file_name} from Google Drive...')\n",
        "        os.remove(zip_path)\n",
        "    else:\n",
        "        print(f'{file_name} not found in Google Drive. Skipping removal.')\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ug2pyCc3jP9g",
        "outputId": "4a438195-f5cd-46af-b35d-82e9bcc1efc6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading ribfrac-train-images-1.zip...\n",
            "--2024-12-24 08:14:07--  https://zenodo.org/records/3893508/files/ribfrac-train-images-1.zip?download=1\n",
            "Resolving zenodo.org (zenodo.org)... 188.185.43.25, 188.185.48.194, 188.185.45.92, ...\n",
            "Connecting to zenodo.org (zenodo.org)|188.185.43.25|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 36605667168 (34G) [application/octet-stream]\n",
            "Saving to: ‘/content/drive/MyDrive/upwork/RibSeg/RibFrac/ribfrac-train-images-1.zip’\n",
            "\n",
            "         /content/d   0%[                    ]  64.81M  1017KB/s    eta 12h 41m^C\n",
            "Downloading ribfrac-train-images-2.zip...\n",
            "--2024-12-24 08:15:33--  https://zenodo.org/records/3893498/files/ribfrac-train-images-2.zip?download=1\n",
            "Resolving zenodo.org (zenodo.org)... 188.185.43.25, 188.185.48.194, 188.185.45.92, ...\n",
            "Connecting to zenodo.org (zenodo.org)|188.185.43.25|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 14562971473 (14G) [application/octet-stream]\n",
            "Saving to: ‘/content/drive/MyDrive/upwork/RibSeg/RibFrac/ribfrac-train-images-2.zip’\n",
            "\n",
            ".zip                  0%[                    ]  15.14M   626KB/s    eta 6h 9m  ^C\n",
            "Downloading ribfrac-val-images.zip...\n",
            "--2024-12-24 08:15:58--  https://zenodo.org/records/3893496/files/ribfrac-val-images.zip?download=1\n",
            "Resolving zenodo.org (zenodo.org)... 188.185.43.25, 188.185.48.194, 188.185.45.92, ...\n",
            "Connecting to zenodo.org (zenodo.org)|188.185.43.25|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 8729000113 (8.1G) [application/octet-stream]\n",
            "Saving to: ‘/content/drive/MyDrive/upwork/RibSeg/RibFrac/ribfrac-val-images.zip’\n",
            "\n",
            "ive/upwork/RibSeg/R   0%[                    ]  39.96M   683KB/s    eta 3h 37m "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Explanation of above code**\n",
        "\n",
        "*   Define the Destination Path:\n",
        "\n",
        "All RibFrac dataset parts will be saved to /content/drive/MyDrive/upwork/RibSeg/RibFrac.\n",
        "\n",
        "*   Download Each Dataset Part:\n",
        "\n",
        "The download_ribfrac function uses wget to download each zip file from the provided URLs.\n",
        "It checks if the file already exists to avoid redundant downloads.\n",
        "\n",
        "*   Extract Each Dataset Part:\n",
        "\n",
        "The extract_zip function extracts each zip file into its respective directory within Colab's filesystem (/content/RibSeg/data/ribfrac/...).\n",
        "It checks if the extraction directory already exists to prevent re-extraction.\n",
        "\n",
        "*   Clean Up:\n",
        "\n",
        "After extraction, the zip files are removed from Google Drive to save space. This step is optional and can be skipped if you prefer to keep the zip files.\n",
        "\n",
        "\n",
        "*Important: Ensure that your Google Drive has sufficient storage space to accommodate the RibFrac dataset (several GBs). If you encounter storage issues, consider upgrading your Google Drive storage or using alternative storage solutions.*"
      ],
      "metadata": {
        "id": "UPXFYBg2j9tW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Data Preparation**\n",
        "\n",
        "Prepare the data for training by running the data_prepare.py script. This script binarizes the CT scans and annotations for rib segmentation.\n",
        "\n"
      ],
      "metadata": {
        "id": "Jbu7h1-VkSFG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Merging the Training Sets**\n",
        "\n",
        "Since the RibFrac Training Set is split into two parts (train_part1 and train_part2), we'll merge them into a single training directory to simplify the data preparation process."
      ],
      "metadata": {
        "id": "MdMpHrCbs8-m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import shutil\n",
        "\n",
        "# Define source directories\n",
        "source_train_part1 = '/content/RibSeg/data/ribfrac/train_part1'\n",
        "source_train_part2 = '/content/RibSeg/data/ribfrac/train_part2'\n",
        "\n",
        "# Define destination directory\n",
        "destination_train = '/content/RibSeg/data/ribfrac/train'\n",
        "\n",
        "# Create the destination directory if it doesn't exist\n",
        "os.makedirs(destination_train, exist_ok=True)\n",
        "\n",
        "# Move all files from train_part1 to train\n",
        "for filename in os.listdir(source_train_part1):\n",
        "    src_file = os.path.join(source_train_part1, filename)\n",
        "    dst_file = os.path.join(destination_train, filename)\n",
        "    shutil.move(src_file, dst_file)\n",
        "\n",
        "# Move all files from train_part2 to train\n",
        "for filename in os.listdir(source_train_part2):\n",
        "    src_file = os.path.join(source_train_part2, filename)\n",
        "    dst_file = os.path.join(destination_train, filename)\n",
        "    shutil.move(src_file, dst_file)\n",
        "\n",
        "# Optionally, remove the now-empty train_part1 and train_part2 directories\n",
        "shutil.rmtree(source_train_part1)\n",
        "shutil.rmtree(source_train_part2)\n",
        "\n",
        "print(\"Merged Training Set Part 1 and Part 2 into a single 'train' directory.\")\n"
      ],
      "metadata": {
        "id": "NeHKn5zbkPuu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Running the Data Preparation Script**\n",
        "\n",
        "Now, execute the data_prepare.py script to preprocess the data."
      ],
      "metadata": {
        "id": "-NWfTXOxtVps"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Ensure you're in the RibSeg directory\n",
        "%cd /content/RibSeg\n",
        "\n",
        "# Run the data preparation script\n",
        "!python data_prepare.py\n"
      ],
      "metadata": {
        "id": "IMbqGPiitW9_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Training the Model**\n",
        "\n",
        "Train the RibSeg V2 model using the provided training script. Training on a GPU significantly speeds up this process."
      ],
      "metadata": {
        "id": "gwpCkuHWtlep"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the directory to save model checkpoints\n",
        "model_directory = '/content/RibSeg/models/ribsegv2_model'\n",
        "\n",
        "# Create the model directory\n",
        "!mkdir -p {model_directory}\n",
        "\n",
        "# Run the training script\n",
        "!python train_ribseg.py --model pointnet2_part_seg_msg --log_dir {model_directory}\n"
      ],
      "metadata": {
        "id": "yzi88GE8tm6t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Monitoring Training**\n",
        "\n",
        "Monitor the training progress through the console logs. Additionally, you can utilize TensorBoard for a more detailed visualization."
      ],
      "metadata": {
        "id": "I8WvoMGbtvY3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load TensorBoard extension\n",
        "%load_ext tensorboard\n",
        "\n",
        "# Launch TensorBoard pointing to the log directory\n",
        "%tensorboard --logdir /content/RibSeg/models/ribsegv2_model\n"
      ],
      "metadata": {
        "id": "4aGQl-lGtxE2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Customizing Training Parameters\n",
        "\n",
        "You can modify training hyperparameters such as learning rate, batch size, and number of epochs by editing the train_ribseg.py script or by adding command-line arguments."
      ],
      "metadata": {
        "id": "NGWKT1RIt38A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Example: Running training with a custom learning rate and number of epochs\n",
        "# !python train_ribseg.py --model pointnet2_part_seg_msg --log_dir {model_directory} --learning_rate 0.001 --epochs 100\n"
      ],
      "metadata": {
        "id": "pFY0F0UJt8t7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Testing and Inference**\n",
        "\n",
        "After training, evaluate the model's performance on the test dataset and perform inference on new CT scans.\n",
        "\n",
        "Testing the Model\n",
        "\n",
        "Evaluate the trained model's performance on the test dataset.\n",
        "\n",
        "**NOTE: test_ribseg.py NOT found in the github repo**"
      ],
      "metadata": {
        "id": "OBUVq4ect-5H"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Run the testing script\n",
        "# !python test_ribseg.py --log_dir /content/RibSeg/models/ribsegv2_model\n"
      ],
      "metadata": {
        "id": "wbOlmjn9uPdX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Running Inference**\n",
        "\n",
        "Perform inference on new CT scans using the trained model."
      ],
      "metadata": {
        "id": "uCl5D_XLuTOq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the directory containing new CT scans for inference\n",
        "inference_input_dir = '/content/RibSeg/data/new_ct_scans'  # Update as necessary\n",
        "\n",
        "# Define the output directory for inference results\n",
        "inference_output_dir = '/content/RibSeg/inference_results'\n",
        "\n",
        "# Create the inference output directory\n",
        "!mkdir -p {inference_output_dir}\n",
        "\n",
        "# Run the inference script\n",
        "!python inference.py --log_dir /content/RibSeg/models/ribsegv2_model --input_dir {inference_input_dir} --output_dir {inference_output_dir}\n"
      ],
      "metadata": {
        "id": "aIugoUlkuUno"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Post-processing**\n",
        "\n",
        "Obtain volume-wise test results by running the post-processing script."
      ],
      "metadata": {
        "id": "cIfuJmsKuhHG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define input and output directories for post-processing\n",
        "post_proc_input_dir = '/content/RibSeg/inference_results'\n",
        "post_proc_output_dir = '/content/RibSeg/final_results'\n",
        "\n",
        "# Create the post-processing output directory\n",
        "!mkdir -p {post_proc_output_dir}\n",
        "\n",
        "# Run the post-processing script\n",
        "!python post_proc.py --input_dir {post_proc_input_dir} --output_dir {post_proc_output_dir}\n"
      ],
      "metadata": {
        "id": "Pr2RGHP0ujrp"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}