files_for_experiments.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DPgIoM45lq5W",
        "outputId": "163a603f-83b6-42f9-d01e-7c9d61ce9732"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Main files:\n",
        "\n",
        "\n",
        "1.   mscoco_40kdecomposed_dependencies_wholeexp.pkl contains the queries for the 40k MSCOCO original images, with decompositions and dependencies. Link: https://drive.google.com/file/d/1GsfeO0qsQN4N9oTNwH9SX_3ctcNFIwAz/view?usp=sharing\n",
        "2.   mscoco_120kdecomposed_wholeexp.pkl contains the queries for the 120k sharegpt4v images, with decompositions at the paragraph, sliding window, and sentence level, but dependencies only at the paragraph and sliding window level. Link: https://drive.google.com/file/d/14OEIZfGr_FhaWUwfuy1Mv7md0NgKLR5W/view?usp=sharing\n",
        "3. Image patch embeddings for the 40k MSCOCO original images. Link: https://drive.google.com/drive/folders/1cFc5Kl_Vn2nAhQDoWYnGky8gvYHvzaIP?usp=sharing\n",
        "4. A function to calculate sentence dependencies for shareGPT4v.\n",
        "\n"
      ],
      "metadata": {
        "id": "GuLIABb36HUW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "MSCOCO original 40k queries"
      ],
      "metadata": {
        "id": "sZL2ugYOz6LN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "img_caption_file_name = '/content/drive/MyDrive/mscoco_40kdecomposed_dependencies_wholeexp.pkl'\n",
        "\n",
        "with open(img_caption_file_name, 'rb') as f:\n",
        "  df_mscoco_40k = pickle.load(f)"
      ],
      "metadata": {
        "id": "1NoS28g_zm-0"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Columns \"caption_triples_ls\" contains the decomposed queries and \"groups\" contain the dependencies for all the 40k mscoco clustered queries."
      ],
      "metadata": {
        "id": "K8hdZlFl0i-r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_mscoco_40k"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 944
        },
        "id": "cIgYPacW0X6Y",
        "outputId": "fb70ee87-4f02-408d-c9cb-7515ecc8cdd8"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "           id             image  \\\n",
              "0           9  000000000009.jpg   \n",
              "1          25  000000000025.jpg   \n",
              "2          30  000000000030.jpg   \n",
              "3          34  000000000034.jpg   \n",
              "4          36  000000000036.jpg   \n",
              "...       ...               ...   \n",
              "43815  343235  000000343235.jpg   \n",
              "43816  271681  000000271681.jpg   \n",
              "43817  268726  000000268726.jpg   \n",
              "43818  478857  000000478857.jpg   \n",
              "43819  467198  000000467198.jpg   \n",
              "\n",
              "                                      caption_sharegpt4v  \\\n",
              "0      In the center of the image, a vibrant blue lun...   \n",
              "1      This image captures a serene moment in a zoo e...   \n",
              "2      The image presents a serene garden scene, cent...   \n",
              "3      This is a detailed description of the image:\\n...   \n",
              "4      In the image, there is a woman standing in fro...   \n",
              "...                                                  ...   \n",
              "43815  In the given image, a tranquil scene unfolds i...   \n",
              "43816  In the heart of a bustling city, a young woman...   \n",
              "43817  This is a black and white photograph capturing...   \n",
              "43818  The image captures a serene suburban scene. Do...   \n",
              "43819  The image captures a small, somewhat neglected...   \n",
              "\n",
              "                                          caption_mscoco  \\\n",
              "0      Closeup of bins of food that include broccoli ...   \n",
              "1        A giraffe eating food from the top of the tree.   \n",
              "2             A flower vase is sitting on a porch stand.   \n",
              "3        A zebra grazing on lush green grass in a field.   \n",
              "4       Woman in swim suit holding parasol on sunny day.   \n",
              "...                                                  ...   \n",
              "43815  Three cats sitting looking out sliding glass d...   \n",
              "43816   A woman holding a quiche in her hand and smiling   \n",
              "43817            A large dog sleeping on top of a couch.   \n",
              "43818  Stop signs on the side of the street near vehi...   \n",
              "43819  A dirty and gross public restroom that needs t...   \n",
              "\n",
              "                                               embedding  \\\n",
              "0      [0.196618, -0.1124176, -0.16870016, 0.03273408...   \n",
              "1      [0.3378602, 0.00365863, -0.32199287, 0.1653291...   \n",
              "2      [-0.034444228, 0.29082185, -0.022862792, -0.42...   \n",
              "3      [-0.12711053, -0.30761, 0.0056521297, -0.01875...   \n",
              "4      [0.2885676, -0.18000656, -0.5650209, 0.0151512...   \n",
              "...                                                  ...   \n",
              "43815  [0.46538836, -0.31321126, -0.32928455, -0.3109...   \n",
              "43816  [-0.022412345, 0.32375985, -0.32947338, -0.019...   \n",
              "43817  [0.3253008, 0.43674475, -0.7003968, 0.13196589...   \n",
              "43818  [0.39890462, -0.24824393, 0.27765816, -0.44707...   \n",
              "43819  [-0.3209242, 0.16469006, 0.29011577, 0.1682263...   \n",
              "\n",
              "                                      caption_triples_ls groups  elapsed_time  \\\n",
              "0      \"Closeup of bins of food.\"|\"Bins of food inclu...  0,1,2     12.546637   \n",
              "1      \"A giraffe.\"|\"Eating food from the top of the ...  0 | 1      9.056211   \n",
              "2           \"A flower vase.\"|\"Sitting on a porch stand.\"  0 | 1      9.195682   \n",
              "3      \"A zebra grazing.\" | \"Lush green grass in a fi...  0 | 1      8.578875   \n",
              "4      \"Woman in swim suit.\" | \"Woman holding parasol...  0,1,2     14.776144   \n",
              "...                                                  ...    ...           ...   \n",
              "43815  \"Three cats sitting looking out sliding glass ...    0,1      4.688795   \n",
              "43816  \"A woman.\"|\"A woman holding a quiche in her ha...  0,1,2      8.097479   \n",
              "43817       \"A large dog.\"|\"Sleeping on top of a couch.\"  0 | 1      4.469425   \n",
              "43818  \"Stop signs on the side of the street.\"|\"Near ...  0,1,2      7.731356   \n",
              "43819  \"A dirty public restroom.\"|\"A gross public res...  0,1,2      8.458007   \n",
              "\n",
              "       num_groups  \n",
              "0               1  \n",
              "1               2  \n",
              "2               2  \n",
              "3               2  \n",
              "4               1  \n",
              "...           ...  \n",
              "43815           1  \n",
              "43816           1  \n",
              "43817           2  \n",
              "43818           1  \n",
              "43819           1  \n",
              "\n",
              "[43820 rows x 9 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-72e00a02-68d3-4e29-b9fa-eb680cb5d8ec\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>image</th>\n",
              "      <th>caption_sharegpt4v</th>\n",
              "      <th>caption_mscoco</th>\n",
              "      <th>embedding</th>\n",
              "      <th>caption_triples_ls</th>\n",
              "      <th>groups</th>\n",
              "      <th>elapsed_time</th>\n",
              "      <th>num_groups</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>9</td>\n",
              "      <td>000000000009.jpg</td>\n",
              "      <td>In the center of the image, a vibrant blue lun...</td>\n",
              "      <td>Closeup of bins of food that include broccoli ...</td>\n",
              "      <td>[0.196618, -0.1124176, -0.16870016, 0.03273408...</td>\n",
              "      <td>\"Closeup of bins of food.\"|\"Bins of food inclu...</td>\n",
              "      <td>0,1,2</td>\n",
              "      <td>12.546637</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>25</td>\n",
              "      <td>000000000025.jpg</td>\n",
              "      <td>This image captures a serene moment in a zoo e...</td>\n",
              "      <td>A giraffe eating food from the top of the tree.</td>\n",
              "      <td>[0.3378602, 0.00365863, -0.32199287, 0.1653291...</td>\n",
              "      <td>\"A giraffe.\"|\"Eating food from the top of the ...</td>\n",
              "      <td>0 | 1</td>\n",
              "      <td>9.056211</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>30</td>\n",
              "      <td>000000000030.jpg</td>\n",
              "      <td>The image presents a serene garden scene, cent...</td>\n",
              "      <td>A flower vase is sitting on a porch stand.</td>\n",
              "      <td>[-0.034444228, 0.29082185, -0.022862792, -0.42...</td>\n",
              "      <td>\"A flower vase.\"|\"Sitting on a porch stand.\"</td>\n",
              "      <td>0 | 1</td>\n",
              "      <td>9.195682</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>34</td>\n",
              "      <td>000000000034.jpg</td>\n",
              "      <td>This is a detailed description of the image:\\n...</td>\n",
              "      <td>A zebra grazing on lush green grass in a field.</td>\n",
              "      <td>[-0.12711053, -0.30761, 0.0056521297, -0.01875...</td>\n",
              "      <td>\"A zebra grazing.\" | \"Lush green grass in a fi...</td>\n",
              "      <td>0 | 1</td>\n",
              "      <td>8.578875</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>36</td>\n",
              "      <td>000000000036.jpg</td>\n",
              "      <td>In the image, there is a woman standing in fro...</td>\n",
              "      <td>Woman in swim suit holding parasol on sunny day.</td>\n",
              "      <td>[0.2885676, -0.18000656, -0.5650209, 0.0151512...</td>\n",
              "      <td>\"Woman in swim suit.\" | \"Woman holding parasol...</td>\n",
              "      <td>0,1,2</td>\n",
              "      <td>14.776144</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43815</th>\n",
              "      <td>343235</td>\n",
              "      <td>000000343235.jpg</td>\n",
              "      <td>In the given image, a tranquil scene unfolds i...</td>\n",
              "      <td>Three cats sitting looking out sliding glass d...</td>\n",
              "      <td>[0.46538836, -0.31321126, -0.32928455, -0.3109...</td>\n",
              "      <td>\"Three cats sitting looking out sliding glass ...</td>\n",
              "      <td>0,1</td>\n",
              "      <td>4.688795</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43816</th>\n",
              "      <td>271681</td>\n",
              "      <td>000000271681.jpg</td>\n",
              "      <td>In the heart of a bustling city, a young woman...</td>\n",
              "      <td>A woman holding a quiche in her hand and smiling</td>\n",
              "      <td>[-0.022412345, 0.32375985, -0.32947338, -0.019...</td>\n",
              "      <td>\"A woman.\"|\"A woman holding a quiche in her ha...</td>\n",
              "      <td>0,1,2</td>\n",
              "      <td>8.097479</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43817</th>\n",
              "      <td>268726</td>\n",
              "      <td>000000268726.jpg</td>\n",
              "      <td>This is a black and white photograph capturing...</td>\n",
              "      <td>A large dog sleeping on top of a couch.</td>\n",
              "      <td>[0.3253008, 0.43674475, -0.7003968, 0.13196589...</td>\n",
              "      <td>\"A large dog.\"|\"Sleeping on top of a couch.\"</td>\n",
              "      <td>0 | 1</td>\n",
              "      <td>4.469425</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43818</th>\n",
              "      <td>478857</td>\n",
              "      <td>000000478857.jpg</td>\n",
              "      <td>The image captures a serene suburban scene. Do...</td>\n",
              "      <td>Stop signs on the side of the street near vehi...</td>\n",
              "      <td>[0.39890462, -0.24824393, 0.27765816, -0.44707...</td>\n",
              "      <td>\"Stop signs on the side of the street.\"|\"Near ...</td>\n",
              "      <td>0,1,2</td>\n",
              "      <td>7.731356</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43819</th>\n",
              "      <td>467198</td>\n",
              "      <td>000000467198.jpg</td>\n",
              "      <td>The image captures a small, somewhat neglected...</td>\n",
              "      <td>A dirty and gross public restroom that needs t...</td>\n",
              "      <td>[-0.3209242, 0.16469006, 0.29011577, 0.1682263...</td>\n",
              "      <td>\"A dirty public restroom.\"|\"A gross public res...</td>\n",
              "      <td>0,1,2</td>\n",
              "      <td>8.458007</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>43820 rows × 9 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-72e00a02-68d3-4e29-b9fa-eb680cb5d8ec')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-72e00a02-68d3-4e29-b9fa-eb680cb5d8ec button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-72e00a02-68d3-4e29-b9fa-eb680cb5d8ec');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-7eca88e2-5864-489a-a7b6-d5d7abd1eed0\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7eca88e2-5864-489a-a7b6-d5d7abd1eed0')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-7eca88e2-5864-489a-a7b6-d5d7abd1eed0 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_bb6c541f-d605-42d9-a527-e723ce548c88\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df_mscoco_40k')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_bb6c541f-d605-42d9-a527-e723ce548c88 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df_mscoco_40k');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df_mscoco_40k",
              "summary": "{\n  \"name\": \"df_mscoco_40k\",\n  \"rows\": 43820,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 170915,\n        \"min\": 9,\n        \"max\": 581921,\n        \"num_unique_values\": 43820,\n        \"samples\": [\n          114954,\n          76846,\n          375245\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"image\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 43820,\n        \"samples\": [\n          \"000000114954.jpg\",\n          \"000000076846.jpg\",\n          \"000000375245.jpg\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"caption_sharegpt4v\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 43818,\n        \"samples\": [\n          \"In the image, a lively scene unfolds on a sports field blanketed in artificial turf. The field, vibrant under the clear blue sky, is marked by a goal post standing tall at one end. A group of people, clad in brightly colored shirts, are engrossed in a game. Their actions are varied and dynamic - one individual is captured in the act of throwing a ball, their body language reflecting the intensity of the moment. Another person is seen running towards the ball, their determination evident in their stride. Yet another stands with arms outstretched, perhaps strategizing their next move or signaling to a teammate. The background is adorned with trees and buildings, adding depth to the scene and indicating that the field is likely situated in a populated area. The image is a snapshot of an active moment, encapsulating the energy and camaraderie inherent in team sports.\",\n          \"In the heart of this image lies a spacious living room, bathed in light from three windows adorned with wooden shutters. The high ceiling gives an airy feel to the room, and a ceiling fan with a light fixture hangs from above, ready to provide a cool breeze.\\n\\nThe room is a symphony of colors and textures. A green sofa invites relaxation, while a black armchair stands nearby, offering another spot to unwind. A blue chair adds a pop of color to the room, and a blue bag rests on the floor next to it.\\n\\nA coffee table sits in the center of the room, holding a vase filled with white flowers that add a touch of nature to the space. A TV is also present, promising entertainment for those who sit on the comfortable furniture.\\n\\nThe floor is covered in tiles, providing a cool and clean surface. A staircase can be seen in the background, leading to the upper levels of the house. The walls are painted white, reflecting the light from the windows and creating a bright and welcoming atmosphere.\",\n          \"In this black and white photograph, a man is the central figure, seated on a bench in a park. He is dressed in a suit and hat, suggesting a formal or semi-formal occasion. In his right hand, he holds a bottle of beer, while his left hand grasps a cane. His face is lit up with a smile, adding a touch of warmth to the scene.\\n\\nThe bench on which he sits is positioned in front of a white picket fence, a classic symbol of suburban life. Behind the man, another person can be seen, but their features are blurred, keeping the focus on the man in the foreground.\\n\\nThe photograph is taken from a slightly elevated angle, giving us a clear view of the man and his surroundings. Despite being in black and white, the image is rich in detail and narrative, inviting us to imagine the story behind it.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"caption_mscoco\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 43380,\n        \"samples\": [\n          \"A man going down a mountain on a snow board.\",\n          \"a man kneeling in the snow with a green snowboard on his feet \",\n          \"The three various sized bags have been placed near each other.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"embedding\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"caption_triples_ls\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 43507,\n        \"samples\": [\n          \"\\\"Men ride motorcycles.\\\"|\\\"The motorcycles have American flags on them.\\\"\",\n          \"\\\"A bag of luggage.\\\"|\\\"In front of a brown couch.\\\"\",\n          \"\\\"A tennis player.\\\"|\\\"In the middle of serving the ball.\\\"\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"groups\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 54,\n        \"samples\": [\n          \"0,1 | 2 | 3\",\n          \"0 | 1,2,3 | 4\",\n          \"0,1,3,4 | 2\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"elapsed_time\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3.5161084135993774,\n        \"min\": 3.814697265625e-06,\n        \"max\": 100.30935025215149,\n        \"num_unique_values\": 41553,\n        \"samples\": [\n          4.382237434387207,\n          5.81282639503479,\n          6.754270553588867\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_groups\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 6,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          1,\n          2,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "ShareGPT4v 120k queries"
      ],
      "metadata": {
        "id": "jKxHgPeP0_Vv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "img_caption_file_name = '/content/drive/MyDrive/mscoco_120kdecomposed_wholeexp.pkl'\n",
        "\n",
        "with open(img_caption_file_name, 'rb') as f:\n",
        "  df_sharegpt4v_120k = pickle.load(f)"
      ],
      "metadata": {
        "id": "61K_fTwh0XzG"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\"paragraph_level\" contains queries decomposed by paragraphs, \"paragraph_slidingwindow_level\" contains queries decomposed by size 2 sliding window (independent across paragraphs) and decomposed by paragraphs, paragraph_slidingwindow_sentence_level contains queries decomposed by sentences, size 2 sliding window, and paragraphs. Same granularities are joined by | and different granularities seperated by #."
      ],
      "metadata": {
        "id": "mp4Cyx-Z2CnJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\"paragraph_level_groups\" contains grouped dependencies for paragraph_level column. \"paragraph_slidingwindow_level_groups\" contains dependencies for paragraph_sliding_window_level column. \"slidingwindow_level_groups\" was used to calculate the intermediate results of just the groups for the sliding window decompositions, so it doesn't have the paragraphs in the groups. If running experiments for paragraph + sliding window, use \"paragraph_slidingwindow_level_groups\" column for dependencies."
      ],
      "metadata": {
        "id": "q7reJx2y2gbZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_sharegpt4v_120k"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 964
        },
        "id": "VG41IdDX2A_U",
        "outputId": "85386cc4-2946-4fb1-8c6b-30dfbe3d0493"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            id             image  \\\n",
              "0            9  000000000009.jpg   \n",
              "1           25  000000000025.jpg   \n",
              "2           30  000000000030.jpg   \n",
              "3           34  000000000034.jpg   \n",
              "4           36  000000000036.jpg   \n",
              "...        ...               ...   \n",
              "118282  265622  000000265622.jpg   \n",
              "118283  478857  000000478857.jpg   \n",
              "118284  467198  000000467198.jpg   \n",
              "118285  302094  000000302094.jpg   \n",
              "118286  408923  000000408923.jpg   \n",
              "\n",
              "                                       caption_sharegpt4v  \\\n",
              "0       In the center of the image, a vibrant blue lun...   \n",
              "1       This image captures a serene moment in a zoo e...   \n",
              "2       The image presents a serene garden scene, cent...   \n",
              "3       This is a detailed description of the image:\\n...   \n",
              "4       In the image, there is a woman standing in fro...   \n",
              "...                                                   ...   \n",
              "118282  In the heart of a cozy living room, two men ar...   \n",
              "118283  The image captures a serene suburban scene. Do...   \n",
              "118284  The image captures a small, somewhat neglected...   \n",
              "118285  The image captures a vibrant scene from a Chin...   \n",
              "118286  The image captures a lively scene on a tennis ...   \n",
              "\n",
              "                                           caption_mscoco  \\\n",
              "0       Closeup of bins of food that include broccoli ...   \n",
              "1         A giraffe eating food from the top of the tree.   \n",
              "2              A flower vase is sitting on a porch stand.   \n",
              "3         A zebra grazing on lush green grass in a field.   \n",
              "4        Woman in swim suit holding parasol on sunny day.   \n",
              "...                                                   ...   \n",
              "118282  Two men in black shirts playing a game with Ni...   \n",
              "118283  Stop signs on the side of the street near vehi...   \n",
              "118284  A dirty and gross public restroom that needs t...   \n",
              "118285       a very nice draw showing a vase with flowers   \n",
              "118286  There are some people playing on a community t...   \n",
              "\n",
              "                                                embedding  \\\n",
              "0       [0.196618, -0.1124176, -0.16870016, 0.03273408...   \n",
              "1       [0.3378602, 0.00365863, -0.32199287, 0.1653291...   \n",
              "2       [-0.034444228, 0.29082185, -0.022862792, -0.42...   \n",
              "3       [-0.12711053, -0.30761, 0.0056521297, -0.01875...   \n",
              "4       [0.2885676, -0.18000656, -0.5650209, 0.0151512...   \n",
              "...                                                   ...   \n",
              "118282  [0.07771664, 0.108801916, -0.4778169, 0.281325...   \n",
              "118283  [0.39890462, -0.24824393, 0.27765816, -0.44707...   \n",
              "118284  [-0.3209242, 0.16469006, 0.29011577, 0.1682263...   \n",
              "118285  [-0.19526501, -0.095813915, -0.23475075, -0.11...   \n",
              "118286  [-0.36754537, -0.18036996, 0.22526744, -0.1092...   \n",
              "\n",
              "        avg_sentences_per_paragraph  \\\n",
              "0                          2.000000   \n",
              "1                          1.750000   \n",
              "2                          3.500000   \n",
              "3                          1.222222   \n",
              "4                          7.000000   \n",
              "...                             ...   \n",
              "118282                     3.333333   \n",
              "118283                     2.600000   \n",
              "118284                     2.250000   \n",
              "118285                     2.250000   \n",
              "118286                     2.666667   \n",
              "\n",
              "                                          paragraph_level  \\\n",
              "0       \"In the center of the image, a vibrant blue lu...   \n",
              "1       \"This image captures a serene moment in a zoo ...   \n",
              "2       \"The image presents a serene garden scene, cen...   \n",
              "3       \"This is a detailed description of the image:\"...   \n",
              "4       \"In the image, there is a woman standing in fr...   \n",
              "...                                                   ...   \n",
              "118282  \"In the heart of a cozy living room, two men a...   \n",
              "118283  \"The image captures a serene suburban scene. D...   \n",
              "118284  \"The image captures a small, somewhat neglecte...   \n",
              "118285  \"The image captures a vibrant scene from a Chi...   \n",
              "118286  \"The image captures a lively scene on a tennis...   \n",
              "\n",
              "                            paragraph_slidingwindow_level  \\\n",
              "0       \"In the center of the image, a vibrant blue lu...   \n",
              "1       \"This image captures a serene moment in a zoo ...   \n",
              "2       \"The image presents a serene garden scene, cen...   \n",
              "3       \"This is a detailed description of the image:\"...   \n",
              "4       \"In the image, there is a woman standing in fr...   \n",
              "...                                                   ...   \n",
              "118282  \"In the heart of a cozy living room, two men a...   \n",
              "118283  \"The image captures a serene suburban scene. D...   \n",
              "118284  \"The image captures a small, somewhat neglecte...   \n",
              "118285  \"The image captures a vibrant scene from a Chi...   \n",
              "118286  \"The image captures a lively scene on a tennis...   \n",
              "\n",
              "                   paragraph_slidingwindow_sentence_level  \\\n",
              "0       \"In the center of the image, a vibrant blue lu...   \n",
              "1       \"This image captures a serene moment in a zoo ...   \n",
              "2       \"The image presents a serene garden scene, cen...   \n",
              "3       \"This is a detailed description of the image:....   \n",
              "4       \"In the image, there is a woman standing in fr...   \n",
              "...                                                   ...   \n",
              "118282  \"In the heart of a cozy living room, two men a...   \n",
              "118283  \"The image captures a serene suburban scene.\"|...   \n",
              "118284  \"The image captures a small, somewhat neglecte...   \n",
              "118285  \"The image captures a vibrant scene from a Chi...   \n",
              "118286  \"The image captures a lively scene on a tennis...   \n",
              "\n",
              "       paragraph_level_groups         paragraph_slidingwindow_level_groups  \\\n",
              "0                 0|1|2|3|4|5                 0|1|2,3|4|5|6|7|8|9|10|11|12   \n",
              "1                     0|1|2|3                              0|1|2|3|4|5|6|7   \n",
              "2                         0|1                                  0,1|2,3|4|5   \n",
              "3           0|1|2|3|4|5|6|7|8  0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17   \n",
              "4                           0                                0,1,2,3,4,5|6   \n",
              "...                       ...                                          ...   \n",
              "118282                  0|1|2                          0,1,2|3,4,5|6|7|8|9   \n",
              "118283              0|1|2|3|4                    0,1|2|3|4,5|6|7|8|9|10|11   \n",
              "118284                0|1|2|3                          0,1,2|3|4|5|6|7|8|9   \n",
              "118285                0|1|2|3                            0,1|2|3|4|5|6|7|8   \n",
              "118286                  0|1|2                            0,1,2|3,4|5|6|7|8   \n",
              "\n",
              "       slidingwindow_level_groups  \n",
              "0                   0|1|2,3|4|5|6  \n",
              "1                         0|1|2|3  \n",
              "2                         0,1|2,3  \n",
              "3               0|1|2|3|4|5|6|7|8  \n",
              "4                     0,1,2,3,4,5  \n",
              "...                           ...  \n",
              "118282              0,1,2|3,4,5|6  \n",
              "118283              0,1|2|3|4,5|6  \n",
              "118284                0,1,2|3|4|5  \n",
              "118285                  0,1|2|3|4  \n",
              "118286                0,1,2|3,4|5  \n",
              "\n",
              "[118287 rows x 12 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3ce36ea9-c643-4429-868c-e2d70f82d418\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>image</th>\n",
              "      <th>caption_sharegpt4v</th>\n",
              "      <th>caption_mscoco</th>\n",
              "      <th>embedding</th>\n",
              "      <th>avg_sentences_per_paragraph</th>\n",
              "      <th>paragraph_level</th>\n",
              "      <th>paragraph_slidingwindow_level</th>\n",
              "      <th>paragraph_slidingwindow_sentence_level</th>\n",
              "      <th>paragraph_level_groups</th>\n",
              "      <th>paragraph_slidingwindow_level_groups</th>\n",
              "      <th>slidingwindow_level_groups</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>9</td>\n",
              "      <td>000000000009.jpg</td>\n",
              "      <td>In the center of the image, a vibrant blue lun...</td>\n",
              "      <td>Closeup of bins of food that include broccoli ...</td>\n",
              "      <td>[0.196618, -0.1124176, -0.16870016, 0.03273408...</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>\"In the center of the image, a vibrant blue lu...</td>\n",
              "      <td>\"In the center of the image, a vibrant blue lu...</td>\n",
              "      <td>\"In the center of the image, a vibrant blue lu...</td>\n",
              "      <td>0|1|2|3|4|5</td>\n",
              "      <td>0|1|2,3|4|5|6|7|8|9|10|11|12</td>\n",
              "      <td>0|1|2,3|4|5|6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>25</td>\n",
              "      <td>000000000025.jpg</td>\n",
              "      <td>This image captures a serene moment in a zoo e...</td>\n",
              "      <td>A giraffe eating food from the top of the tree.</td>\n",
              "      <td>[0.3378602, 0.00365863, -0.32199287, 0.1653291...</td>\n",
              "      <td>1.750000</td>\n",
              "      <td>\"This image captures a serene moment in a zoo ...</td>\n",
              "      <td>\"This image captures a serene moment in a zoo ...</td>\n",
              "      <td>\"This image captures a serene moment in a zoo ...</td>\n",
              "      <td>0|1|2|3</td>\n",
              "      <td>0|1|2|3|4|5|6|7</td>\n",
              "      <td>0|1|2|3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>30</td>\n",
              "      <td>000000000030.jpg</td>\n",
              "      <td>The image presents a serene garden scene, cent...</td>\n",
              "      <td>A flower vase is sitting on a porch stand.</td>\n",
              "      <td>[-0.034444228, 0.29082185, -0.022862792, -0.42...</td>\n",
              "      <td>3.500000</td>\n",
              "      <td>\"The image presents a serene garden scene, cen...</td>\n",
              "      <td>\"The image presents a serene garden scene, cen...</td>\n",
              "      <td>\"The image presents a serene garden scene, cen...</td>\n",
              "      <td>0|1</td>\n",
              "      <td>0,1|2,3|4|5</td>\n",
              "      <td>0,1|2,3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>34</td>\n",
              "      <td>000000000034.jpg</td>\n",
              "      <td>This is a detailed description of the image:\\n...</td>\n",
              "      <td>A zebra grazing on lush green grass in a field.</td>\n",
              "      <td>[-0.12711053, -0.30761, 0.0056521297, -0.01875...</td>\n",
              "      <td>1.222222</td>\n",
              "      <td>\"This is a detailed description of the image:\"...</td>\n",
              "      <td>\"This is a detailed description of the image:\"...</td>\n",
              "      <td>\"This is a detailed description of the image:....</td>\n",
              "      <td>0|1|2|3|4|5|6|7|8</td>\n",
              "      <td>0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17</td>\n",
              "      <td>0|1|2|3|4|5|6|7|8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>36</td>\n",
              "      <td>000000000036.jpg</td>\n",
              "      <td>In the image, there is a woman standing in fro...</td>\n",
              "      <td>Woman in swim suit holding parasol on sunny day.</td>\n",
              "      <td>[0.2885676, -0.18000656, -0.5650209, 0.0151512...</td>\n",
              "      <td>7.000000</td>\n",
              "      <td>\"In the image, there is a woman standing in fr...</td>\n",
              "      <td>\"In the image, there is a woman standing in fr...</td>\n",
              "      <td>\"In the image, there is a woman standing in fr...</td>\n",
              "      <td>0</td>\n",
              "      <td>0,1,2,3,4,5|6</td>\n",
              "      <td>0,1,2,3,4,5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>118282</th>\n",
              "      <td>265622</td>\n",
              "      <td>000000265622.jpg</td>\n",
              "      <td>In the heart of a cozy living room, two men ar...</td>\n",
              "      <td>Two men in black shirts playing a game with Ni...</td>\n",
              "      <td>[0.07771664, 0.108801916, -0.4778169, 0.281325...</td>\n",
              "      <td>3.333333</td>\n",
              "      <td>\"In the heart of a cozy living room, two men a...</td>\n",
              "      <td>\"In the heart of a cozy living room, two men a...</td>\n",
              "      <td>\"In the heart of a cozy living room, two men a...</td>\n",
              "      <td>0|1|2</td>\n",
              "      <td>0,1,2|3,4,5|6|7|8|9</td>\n",
              "      <td>0,1,2|3,4,5|6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>118283</th>\n",
              "      <td>478857</td>\n",
              "      <td>000000478857.jpg</td>\n",
              "      <td>The image captures a serene suburban scene. Do...</td>\n",
              "      <td>Stop signs on the side of the street near vehi...</td>\n",
              "      <td>[0.39890462, -0.24824393, 0.27765816, -0.44707...</td>\n",
              "      <td>2.600000</td>\n",
              "      <td>\"The image captures a serene suburban scene. D...</td>\n",
              "      <td>\"The image captures a serene suburban scene. D...</td>\n",
              "      <td>\"The image captures a serene suburban scene.\"|...</td>\n",
              "      <td>0|1|2|3|4</td>\n",
              "      <td>0,1|2|3|4,5|6|7|8|9|10|11</td>\n",
              "      <td>0,1|2|3|4,5|6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>118284</th>\n",
              "      <td>467198</td>\n",
              "      <td>000000467198.jpg</td>\n",
              "      <td>The image captures a small, somewhat neglected...</td>\n",
              "      <td>A dirty and gross public restroom that needs t...</td>\n",
              "      <td>[-0.3209242, 0.16469006, 0.29011577, 0.1682263...</td>\n",
              "      <td>2.250000</td>\n",
              "      <td>\"The image captures a small, somewhat neglecte...</td>\n",
              "      <td>\"The image captures a small, somewhat neglecte...</td>\n",
              "      <td>\"The image captures a small, somewhat neglecte...</td>\n",
              "      <td>0|1|2|3</td>\n",
              "      <td>0,1,2|3|4|5|6|7|8|9</td>\n",
              "      <td>0,1,2|3|4|5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>118285</th>\n",
              "      <td>302094</td>\n",
              "      <td>000000302094.jpg</td>\n",
              "      <td>The image captures a vibrant scene from a Chin...</td>\n",
              "      <td>a very nice draw showing a vase with flowers</td>\n",
              "      <td>[-0.19526501, -0.095813915, -0.23475075, -0.11...</td>\n",
              "      <td>2.250000</td>\n",
              "      <td>\"The image captures a vibrant scene from a Chi...</td>\n",
              "      <td>\"The image captures a vibrant scene from a Chi...</td>\n",
              "      <td>\"The image captures a vibrant scene from a Chi...</td>\n",
              "      <td>0|1|2|3</td>\n",
              "      <td>0,1|2|3|4|5|6|7|8</td>\n",
              "      <td>0,1|2|3|4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>118286</th>\n",
              "      <td>408923</td>\n",
              "      <td>000000408923.jpg</td>\n",
              "      <td>The image captures a lively scene on a tennis ...</td>\n",
              "      <td>There are some people playing on a community t...</td>\n",
              "      <td>[-0.36754537, -0.18036996, 0.22526744, -0.1092...</td>\n",
              "      <td>2.666667</td>\n",
              "      <td>\"The image captures a lively scene on a tennis...</td>\n",
              "      <td>\"The image captures a lively scene on a tennis...</td>\n",
              "      <td>\"The image captures a lively scene on a tennis...</td>\n",
              "      <td>0|1|2</td>\n",
              "      <td>0,1,2|3,4|5|6|7|8</td>\n",
              "      <td>0,1,2|3,4|5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>118287 rows × 12 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3ce36ea9-c643-4429-868c-e2d70f82d418')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-3ce36ea9-c643-4429-868c-e2d70f82d418 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-3ce36ea9-c643-4429-868c-e2d70f82d418');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-e1cc84aa-2675-4572-9724-e67a865ef1c4\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e1cc84aa-2675-4572-9724-e67a865ef1c4')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-e1cc84aa-2675-4572-9724-e67a865ef1c4 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_a95720f1-5d8f-4ec5-9c3b-1fa931353023\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df_sharegpt4v_120k')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_a95720f1-5d8f-4ec5-9c3b-1fa931353023 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df_sharegpt4v_120k');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df_sharegpt4v_120k"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Image patch embeddings for 40k MSCOCO original images"
      ],
      "metadata": {
        "id": "2jkap3u23sbD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "File paths in drive folder share with you:"
      ],
      "metadata": {
        "id": "YkY8NaQb39mK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "/content/drive/MyDrive/embeddings_for_40k/saved_img_embs_slic_40k_710db392ea0a717e1c873e6b9bc8959ecef704b5399893d64aca9f4492ac6396.pkl"
      ],
      "metadata": {
        "id": "ya9Rq3Kz4CRn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "/content/drive/MyDrive/embeddings_for_40k/saved_patches_slic_4_40k_710db392ea0a717e1c873e6b9bc8959ecef704b5399893d64aca9f4492ac6396.pkl"
      ],
      "metadata": {
        "id": "yCKXkqnD4Jsi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "/content/drive/MyDrive/embeddings_for_40k/saved_patches_slic_8_40k_710db392ea0a717e1c873e6b9bc8959ecef704b5399893d64aca9f4492ac6396.pkl"
      ],
      "metadata": {
        "id": "0ptQUSvb4KvE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "/content/drive/MyDrive/embeddings_for_40k/saved_patches_slic_16_40k_710db392ea0a717e1c873e6b9bc8959ecef704b5399893d64aca9f4492ac6396.pkl"
      ],
      "metadata": {
        "id": "WHYfNFQN4M8a"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "/content/drive/MyDrive/embeddings_for_40k/saved_patches_slic_64_40k_710db392ea0a717e1c873e6b9bc8959ecef704b5399893d64aca9f4492ac6396.pkl"
      ],
      "metadata": {
        "id": "YPv7chI-4Oen"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "/content/drive/MyDrive/embeddings_for_40k/saved_patches_slic_128_40k_710db392ea0a717e1c873e6b9bc8959ecef704b5399893d64aca9f4492ac6396.pkl"
      ],
      "metadata": {
        "id": "OwafFyKW4Qc9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Function for generating sentence dependencies for ShareGPT4v:"
      ],
      "metadata": {
        "id": "MyXPm-yu5SAd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_sentence_decomp(sentence):\n",
        "  return sentence.split('#')[0]"
      ],
      "metadata": {
        "id": "LVPcdk165WDO"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import nest_asyncio\n",
        "import asyncio\n",
        "import aiohttp\n",
        "import ssl\n",
        "import certifi\n",
        "import time\n",
        "import pandas as pd\n",
        "import pickle\n",
        "\n",
        "nest_asyncio.apply()\n",
        "\n",
        "# Settings\n",
        "MODEL = \"gpt-3.5-turbo\"\n",
        "OPENAI_SECRET_KEY = \"sk-proj-X...X\"\n",
        "MAX_RETRIES = 5  # Maximum number of retries\n",
        "BATCH_SIZE = 100\n",
        "\n",
        "class UnionFind:\n",
        "    def __init__(self, size):\n",
        "        self.parent = list(range(size))\n",
        "        self.rank = [1] * size\n",
        "\n",
        "    def find(self, p):\n",
        "        if self.parent[p] != p:\n",
        "            self.parent[p] = self.find(self.parent[p])  # Path compression\n",
        "        return self.parent[p]\n",
        "\n",
        "    def union(self, p, q):\n",
        "        rootP = self.find(p)\n",
        "        rootQ = self.find(q)\n",
        "        if rootP != rootQ:\n",
        "            # Union by rank\n",
        "            if self.rank[rootP] > self.rank[rootQ]:\n",
        "                self.parent[rootQ] = rootP\n",
        "            elif self.rank[rootP] < self.rank[rootQ]:\n",
        "                self.parent[rootP] = rootQ\n",
        "            else:\n",
        "                self.parent[rootQ] = rootP\n",
        "                self.rank[rootP] += 1\n",
        "\n",
        "# Asynchronous function to check dependency\n",
        "async def check_dependency_async(session, sentence, segment1, segment2, retries=MAX_RETRIES):\n",
        "    payload = {\n",
        "        'model': MODEL,\n",
        "        'messages': [\n",
        "          {\"role\": \"system\", \"content\": \"Define two phrases as dependent when the two phrases share an explicitly defined physical and material object between them. Given two phrases from the passage, output whether the two phrases are dependent. You may answer in only one digit, 1 meaning dependent, 0 meaning independent. Only output 1 if you are 100% certain. Do not consider any vague dependencies.\"},\n",
        "          {\"role\": \"user\", \"content\": f\"Two phrases: {segment1}, {segment2}\\nPlease output 0 or 1, 1 meaning they are dependent phrases, 0 meaning they are independent phrases.\"}\n",
        "        ],\n",
        "        'max_tokens': 1,\n",
        "        'n': 1,\n",
        "        'temperature': 0\n",
        "    }\n",
        "    for attempt in range(retries):\n",
        "        try:\n",
        "            async with session.post(\n",
        "                url='https://api.openai.com/v1/chat/completions',\n",
        "                headers={\"Content-Type\": \"application/json\", \"Authorization\": f\"Bearer {OPENAI_SECRET_KEY}\"},\n",
        "                json=payload,\n",
        "                ssl=ssl.create_default_context(cafile=certifi.where())\n",
        "            ) as response:\n",
        "                response_json = await response.json()\n",
        "            if \"error\" in response_json:\n",
        "                print(f\"OpenAI request failed with error {response_json['error']}\")\n",
        "                if response_json['error']['code'] == 'rate_limit_exceeded':\n",
        "                    retry_after = 10\n",
        "                    await asyncio.sleep(retry_after)  # Wait for the retry-after duration before retrying\n",
        "                continue\n",
        "            result = response_json['choices'][0]['message']['content']\n",
        "            if result in {'1', '0'}:\n",
        "                print(result)\n",
        "                return int(result)\n",
        "            else:\n",
        "                print(\"errors\")\n",
        "                return 0\n",
        "        except Exception as e:\n",
        "            print(f\"Request failed: {e}\")\n",
        "    print(\"errors\")\n",
        "    return 0\n",
        "\n",
        "# Asynchronous function to group dependent segments for a single sentence\n",
        "async def group_dependent_segments_async(session, sentence, segments):\n",
        "    start_time = time.time()  # Start time\n",
        "    segments_list = segments.split('|')\n",
        "    n = len(segments_list)\n",
        "\n",
        "    # Initialize Union-Find\n",
        "    uf = UnionFind(n)\n",
        "\n",
        "    # Perform pairwise comparisons sequentially\n",
        "    for i in range(n):\n",
        "        for j in range(i + 1, n):\n",
        "            if uf.find(i) != uf.find(j):  # Check if i and j are already in the same group\n",
        "                result = await check_dependency_async(session, sentence, segments_list[i], segments_list[j])\n",
        "                if result == 1:\n",
        "                    uf.union(i, j)\n",
        "                await asyncio.sleep(2)  # Small delay to avoid hitting limits too quickly\n",
        "            else:\n",
        "              print('already in same group')\n",
        "\n",
        "    # Group segments based on union-find results\n",
        "    groups = {}\n",
        "    for i in range(n):\n",
        "        root = uf.find(i)\n",
        "        if root not in groups:\n",
        "            groups[root] = []\n",
        "        groups[root].append(i)\n",
        "\n",
        "    # Convert indices back to segments\n",
        "    grouped_indices = [\",\".join(map(str, sorted(group))) for group in groups.values()]\n",
        "    end_time = time.time()  # End time\n",
        "    elapsed_time = end_time - start_time  # Calculate elapsed time\n",
        "    return \" | \".join(grouped_indices), elapsed_time\n",
        "\n",
        "# Asynchronous function to process multiple sentences in parallel\n",
        "async def process_sentences_in_bulk(sentences_and_segments):\n",
        "    async with aiohttp.ClientSession() as session:\n",
        "        results = []\n",
        "        for i in range(0, len(sentences_and_segments), BATCH_SIZE):\n",
        "            batch = sentences_and_segments[i:i+BATCH_SIZE]\n",
        "            tasks = [group_dependent_segments_async(session, sentence, segments) for sentence, segments in batch]\n",
        "            batch_results = await asyncio.gather(*tasks)\n",
        "            results.extend(batch_results)\n",
        "            if i + BATCH_SIZE < len(sentences_and_segments):\n",
        "                await asyncio.sleep(10)  # Wait for 60 seconds between batches to respect rate limits\n",
        "            print(\"Processed: \" + str(i))\n",
        "    return results\n",
        "\n",
        "def apply_to_dataframe(df, sentence_col, segments_col):\n",
        "    # Create a copy of the DataFrame to avoid SettingWithCopyWarning\n",
        "    df_copy = df.copy()\n",
        "    sentences_and_segments = list(zip(df_copy[sentence_col], df_copy[segments_col]))\n",
        "    results = asyncio.run(process_sentences_in_bulk(sentences_and_segments))\n",
        "    df_copy['sentence_level_groups'] = [result[0] for result in results]\n",
        "    df_copy['elapsed_time'] = [result[1] for result in results]\n",
        "    return df_copy\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    begin = time.time()\n",
        "    df = df_sharegpt4v_120k[0:1].copy()\n",
        "    df['sentence_decompositions'] = df['paragraph_slidingwindow_sentence_level'].apply(lambda x: get_sentence_decomp(x))\n",
        "    final_df = apply_to_dataframe(df, 'caption_sharegpt4v', 'sentence_decompositions')\n",
        "    end = time.time()\n",
        "    total_time = end - begin\n",
        "    print(f\"Operation took a total of {total_time} seconds.\")\n",
        "\n",
        "    pickle_file_path = 'INSERT_HERE'\n",
        "    with open(pickle_file_path, 'wb') as f:\n",
        "      pickle.dump(final_df, f)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NeA5a2UTmeOA",
        "outputId": "f85370b7-6e07-4ef8-f4e9-2051455c81a6"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "already in same group\n",
            "Processed: 0\n",
            "Operation took a total of 27.044411420822144 seconds.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "DIxSW-LW5Cua"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}