main.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4MpRz48IHcdf"
      },
      "source": [
        "# Information Retrieval project\n",
        "\n",
        "**Authors:** L.Arduini, D.N.Ghaneh, L.Menchini, C.Petruzzella"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0TJLe-DSu53"
      },
      "source": [
        "## Abstract\n",
        "This project focuses on developing and evaluating a custom Information Retrieval (IR) pipeline. The MSMARCO Passage dataset undergoes preprocessing steps such as tokenization, stemming, and stopword removal using NLTK.\n",
        "\n",
        "An inverted index is implemented to store terms, enabling BM25 and TF-IDF ranking through both DAAT and TAAT approaches. Retrieval performance is evaluated using the ir_measures library for TREC-style assessments, leveraging TREC 2020 queries and QRELs. Additionally, PyTerrier is used for benchmarking the retrieval effectiveness.\n",
        "\n",
        "The goal is to showcase a complete IR workflow, from text normalization to performance comparison with established tools.\n",
        "\n",
        "**Dataset:** The [MSMARCO Passage dataset](https://ir-datasets.com/msmarco-passage.html), consisting of 8,841,823 real-world web passages from diverse sources, offers substantial variety, making it ideal for training and evaluating retrieval models under realistic conditions.\n",
        "\n",
        "**Evaluation:** For evaluation the trec-2020-dl dataset has been used."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KR2_yM6TZuWF"
      },
      "source": [
        "## Instructions to Run\n",
        "We recommend running the project on Google Colab. In particular, we suggest selecting a runtime with GPU support, such as the \"v2-8 TPU\", to access better computational resources, which will result in reduced execution time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lRbriC4Brwi4"
      },
      "source": [
        "# 0. Setup environment and dependencies\n",
        "This section ensures that all necessary packages are installed and loaded.\n",
        "\n",
        "**Note:** The project uses `ir_datasets`, `nltk`, and `ir_measures`, along with several utilities for processing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vpWABC3lHcdg",
        "outputId": "54347c3e-7cf5-425e-f2d9-52288d6f584a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting ir_datasets\n",
            "  Downloading ir_datasets-0.5.9-py3-none-any.whl.metadata (12 kB)\n",
            "Requirement already satisfied: beautifulsoup4>=4.4.1 in /usr/local/lib/python3.10/dist-packages (from ir_datasets) (4.12.3)\n",
            "Collecting inscriptis>=2.2.0 (from ir_datasets)\n",
            "  Downloading inscriptis-2.5.0-py3-none-any.whl.metadata (25 kB)\n",
            "Collecting lxml>=4.5.2 (from ir_datasets)\n",
            "  Downloading lxml-5.3.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.8 kB)\n",
            "Requirement already satisfied: numpy>=1.18.1 in /usr/local/lib/python3.10/dist-packages (from ir_datasets) (1.26.4)\n",
            "Requirement already satisfied: pyyaml>=5.3.1 in /usr/local/lib/python3.10/dist-packages (from ir_datasets) (6.0.2)\n",
            "Requirement already satisfied: requests>=2.22.0 in /usr/local/lib/python3.10/dist-packages (from ir_datasets) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from ir_datasets) (4.67.1)\n",
            "Collecting trec-car-tools>=2.5.4 (from ir_datasets)\n",
            "  Downloading trec_car_tools-2.6-py3-none-any.whl.metadata (640 bytes)\n",
            "Collecting lz4>=3.1.10 (from ir_datasets)\n",
            "  Downloading lz4-4.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.7 kB)\n",
            "Collecting warc3-wet>=0.2.3 (from ir_datasets)\n",
            "  Downloading warc3_wet-0.2.5-py3-none-any.whl.metadata (2.2 kB)\n",
            "Collecting warc3-wet-clueweb09>=0.2.5 (from ir_datasets)\n",
            "  Downloading warc3-wet-clueweb09-0.2.5.tar.gz (17 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting zlib-state>=0.1.3 (from ir_datasets)\n",
            "  Downloading zlib_state-0.1.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.3 kB)\n",
            "Collecting ijson>=3.1.3 (from ir_datasets)\n",
            "  Downloading ijson-3.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)\n",
            "Collecting unlzw3>=0.2.1 (from ir_datasets)\n",
            "  Downloading unlzw3-0.2.3-py3-none-any.whl.metadata (2.3 kB)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4>=4.4.1->ir_datasets) (2.6)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.22.0->ir_datasets) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.22.0->ir_datasets) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.22.0->ir_datasets) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.22.0->ir_datasets) (2024.12.14)\n",
            "Collecting cbor>=1.0.0 (from trec-car-tools>=2.5.4->ir_datasets)\n",
            "  Downloading cbor-1.0.0.tar.gz (20 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Downloading ir_datasets-0.5.9-py3-none-any.whl (347 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m347.9/347.9 kB\u001b[0m \u001b[31m1.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading ijson-3.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.5/114.5 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading inscriptis-2.5.0-py3-none-any.whl (45 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.4/45.4 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading lxml-5.3.0-cp310-cp310-manylinux_2_28_x86_64.whl (5.0 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.0/5.0 MB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading lz4-4.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m15.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading trec_car_tools-2.6-py3-none-any.whl (8.4 kB)\n",
            "Downloading unlzw3-0.2.3-py3-none-any.whl (6.7 kB)\n",
            "Downloading warc3_wet-0.2.5-py3-none-any.whl (18 kB)\n",
            "Downloading zlib_state-0.1.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (21 kB)\n",
            "Building wheels for collected packages: warc3-wet-clueweb09, cbor\n",
            "  Building wheel for warc3-wet-clueweb09 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for warc3-wet-clueweb09: filename=warc3_wet_clueweb09-0.2.5-py3-none-any.whl size=18920 sha256=7ed84723d885b2e65297c3e132d8b39b75e8cbee8d80106c9e476f812650480d\n",
            "  Stored in directory: /root/.cache/pip/wheels/1a/d7/91/7ffb991df87e62355d945745035470ba2616aa3d83a250b5f9\n",
            "  Building wheel for cbor (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for cbor: filename=cbor-1.0.0-cp310-cp310-linux_x86_64.whl size=53434 sha256=995029094ccff474119078c4bb8f2794b3b4fe49a683f4dca362c925f434014d\n",
            "  Stored in directory: /root/.cache/pip/wheels/85/df/c9/b39e40eccaf76dbd218556639a6dc81562226f4c6a64902c85\n",
            "Successfully built warc3-wet-clueweb09 cbor\n",
            "Installing collected packages: warc3-wet-clueweb09, warc3-wet, ijson, cbor, zlib-state, unlzw3, trec-car-tools, lz4, lxml, inscriptis, ir_datasets\n",
            "Successfully installed cbor-1.0.0 ijson-3.3.0 inscriptis-2.5.0 ir_datasets-0.5.9 lxml-5.3.0 lz4-4.3.3 trec-car-tools-2.6 unlzw3-0.2.3 warc3-wet-0.2.5 warc3-wet-clueweb09-0.2.5 zlib-state-0.1.9\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (3.9.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk) (8.1.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk) (1.4.2)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk) (2024.11.6)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nltk) (4.67.1)\n",
            "Collecting ir_measures\n",
            "  Downloading ir_measures-0.3.6-py3-none-any.whl.metadata (7.0 kB)\n",
            "Collecting pytrec-eval-terrier>=0.5.5 (from ir_measures)\n",
            "  Downloading pytrec_eval_terrier-0.5.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (777 bytes)\n",
            "Downloading ir_measures-0.3.6-py3-none-any.whl (60 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.1/60.1 kB\u001b[0m \u001b[31m2.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pytrec_eval_terrier-0.5.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (287 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m287.4/287.4 kB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: pytrec-eval-terrier, ir_measures\n",
            "Successfully installed ir_measures-0.3.6 pytrec-eval-terrier-0.5.6\n",
            "Collecting PyStemmer\n",
            "  Downloading PyStemmer-2.2.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)\n",
            "Downloading PyStemmer-2.2.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (646 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m646.7/646.7 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: PyStemmer\n",
            "Successfully installed PyStemmer-2.2.0.3\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (2.2.2)\n",
            "Requirement already satisfied: numpy>=1.22.4 in /usr/local/lib/python3.10/dist-packages (from pandas) (1.26.4)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.9.0.post0)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas) (2024.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)\n",
            "Collecting python-terrier\n",
            "  Downloading python_terrier-0.12.1-py3-none-any.whl.metadata (11 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from python-terrier) (1.26.4)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from python-terrier) (2.2.2)\n",
            "Requirement already satisfied: more-itertools in /usr/lib/python3/dist-packages (from python-terrier) (8.10.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from python-terrier) (4.67.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from python-terrier) (2.32.3)\n",
            "Requirement already satisfied: ir-datasets>=0.3.2 in /usr/local/lib/python3.10/dist-packages (from python-terrier) (0.5.9)\n",
            "Collecting wget (from python-terrier)\n",
            "  Downloading wget-3.2.zip (10 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting pyjnius>=1.4.2 (from python-terrier)\n",
            "  Downloading pyjnius-1.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)\n",
            "Collecting deprecated (from python-terrier)\n",
            "  Downloading Deprecated-1.2.15-py2.py3-none-any.whl.metadata (5.5 kB)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from python-terrier) (1.13.1)\n",
            "Requirement already satisfied: ir-measures>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from python-terrier) (0.3.6)\n",
            "Requirement already satisfied: pytrec-eval-terrier>=0.5.3 in /usr/local/lib/python3.10/dist-packages (from python-terrier) (0.5.6)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from python-terrier) (3.1.4)\n",
            "Collecting statsmodels (from python-terrier)\n",
            "  Downloading statsmodels-0.14.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.2 kB)\n",
            "Collecting dill (from python-terrier)\n",
            "  Downloading dill-0.3.9-py3-none-any.whl.metadata (10 kB)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from python-terrier) (1.4.2)\n",
            "Collecting chest (from python-terrier)\n",
            "  Downloading chest-0.2.3.tar.gz (9.6 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: beautifulsoup4>=4.4.1 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (4.12.3)\n",
            "Requirement already satisfied: inscriptis>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (2.5.0)\n",
            "Requirement already satisfied: lxml>=4.5.2 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (5.3.0)\n",
            "Requirement already satisfied: pyyaml>=5.3.1 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (6.0.2)\n",
            "Requirement already satisfied: trec-car-tools>=2.5.4 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (2.6)\n",
            "Requirement already satisfied: lz4>=3.1.10 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (4.3.3)\n",
            "Requirement already satisfied: warc3-wet>=0.2.3 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (0.2.5)\n",
            "Requirement already satisfied: warc3-wet-clueweb09>=0.2.5 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (0.2.5)\n",
            "Requirement already satisfied: zlib-state>=0.1.3 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (0.1.9)\n",
            "Requirement already satisfied: ijson>=3.1.3 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (3.3.0)\n",
            "Requirement already satisfied: unlzw3>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from ir-datasets>=0.3.2->python-terrier) (0.2.3)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->python-terrier) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->python-terrier) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->python-terrier) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->python-terrier) (2024.12.14)\n",
            "Collecting heapdict (from chest->python-terrier)\n",
            "  Downloading HeapDict-1.0.1-py3-none-any.whl.metadata (1.9 kB)\n",
            "Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated->python-terrier) (1.14.1)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->python-terrier) (3.0.2)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->python-terrier) (2.9.0.post0)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->python-terrier) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->python-terrier) (2024.2)\n",
            "Collecting patsy>=0.5.6 (from statsmodels->python-terrier)\n",
            "  Downloading patsy-1.0.1-py2.py3-none-any.whl.metadata (3.3 kB)\n",
            "Requirement already satisfied: packaging>=21.3 in /usr/local/lib/python3.10/dist-packages (from statsmodels->python-terrier) (24.2)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4>=4.4.1->ir-datasets>=0.3.2->python-terrier) (2.6)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->python-terrier) (1.17.0)\n",
            "Requirement already satisfied: cbor>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from trec-car-tools>=2.5.4->ir-datasets>=0.3.2->python-terrier) (1.0.0)\n",
            "Downloading python_terrier-0.12.1-py3-none-any.whl (147 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m147.8/147.8 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pyjnius-1.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m28.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading Deprecated-1.2.15-py2.py3-none-any.whl (9.9 kB)\n",
            "Downloading dill-0.3.9-py3-none-any.whl (119 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m119.4/119.4 kB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading statsmodels-0.14.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.8/10.8 MB\u001b[0m \u001b[31m126.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading patsy-1.0.1-py2.py3-none-any.whl (232 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m232.9/232.9 kB\u001b[0m \u001b[31m17.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading HeapDict-1.0.1-py3-none-any.whl (3.9 kB)\n",
            "Building wheels for collected packages: chest, wget\n",
            "  Building wheel for chest (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for chest: filename=chest-0.2.3-py3-none-any.whl size=7611 sha256=935c35dde39c96f73ea88582bc17b56c134a31f41f6f97158e0d225c2c278872\n",
            "  Stored in directory: /root/.cache/pip/wheels/88/cf/99/4773b31f855f9ecedc32a0ae400f7a4a3001b37c439b6d1a73\n",
            "  Building wheel for wget (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9656 sha256=928d2105d448ee4cbd2f8c148e0c666797cbdbf8fefb00efc278355036fd8940\n",
            "  Stored in directory: /root/.cache/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769\n",
            "Successfully built chest wget\n",
            "Installing collected packages: wget, pyjnius, heapdict, patsy, dill, deprecated, chest, statsmodels, python-terrier\n",
            "Successfully installed chest-0.2.3 deprecated-1.2.15 dill-0.3.9 heapdict-1.0.1 patsy-1.0.1 pyjnius-1.6.1 python-terrier-0.12.1 statsmodels-0.14.4 wget-3.2\n",
            "Requirement already satisfied: gdown in /usr/local/lib/python3.10/dist-packages (5.2.0)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from gdown) (4.12.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from gdown) (3.16.1)\n",
            "Requirement already satisfied: requests[socks] in /usr/local/lib/python3.10/dist-packages (from gdown) (2.32.3)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from gdown) (4.67.1)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->gdown) (2.6)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (2024.12.14)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (1.7.1)\n"
          ]
        }
      ],
      "source": [
        "!pip install ir_datasets\n",
        "!pip install nltk\n",
        "!pip install ir_measures\n",
        "!pip install PyStemmer\n",
        "!pip install pandas\n",
        "!pip install python-terrier\n",
        "!pip install --upgrade gdown"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "XPLeMYtBrwi6"
      },
      "outputs": [],
      "source": [
        "import ir_datasets\n",
        "import ir_measures\n",
        "from ir_measures import *\n",
        "import random\n",
        "import re\n",
        "import string\n",
        "import nltk\n",
        "import time\n",
        "from collections import Counter, defaultdict\n",
        "from tqdm.auto import tqdm\n",
        "import gzip\n",
        "import pickle\n",
        "import os\n",
        "import heapq\n",
        "import math\n",
        "import pyterrier as pt\n",
        "from google.colab import drive\n",
        "import os\n",
        "import shutil"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DWWEeE_crwi7"
      },
      "source": [
        "# 1. Loading the dataset\n",
        "\n",
        "This notebook will load the MS MARCO Passage dataset, a standard dataset for Information Retrieval tasks.\n",
        "It contains passages from various sources and is used to train and evaluate retrieval models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "U79MpmqyHcdi"
      },
      "outputs": [],
      "source": [
        "dataset = ir_datasets.load(\"msmarco-passage\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e-0B_CtPSu58"
      },
      "source": [
        "Let’s now import some files from a folder saved in Google Drive. Specifically, we will import the following files: `lexicon`, `inverted indexes`, and `document indexes`.\n",
        "\n",
        "Why are we importing them instead of computing them ourselves by running the code? We’ll see the reasons later, but in short, it’s to save the reader from a long wait."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vLr7vahsEAcs",
        "outputId": "18f016e6-2ef0-4c58-b103-e9c3810f9d28"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n",
            "Retrieving folder contents\n",
            "Processing file 1-6YUREBaAr5H2TNd4v66feF7-gXtRSTB document_index.pickle.gz\n",
            "Processing file 1-6QY48jBNOrtppPDAV80EemnrN-f06yV inverted_file.pickle.gz\n",
            "Processing file 1-5JPLr2Ug3CTAeoYePwOui8CljX5LpV0 lexicon.pickle.gz\n",
            "Processing file 1-L0CzWJPrzmllB-3P5dE1qkV9cgyJL7G stats.pickle.gz\n",
            "Processing file 1-KA42mZpr_KT5E4Jl3isSThNnu2ueUg5 trec_eval_bm25_run_file.txt\n",
            "Processing file 1-Aj1Q513p315ymq_bgDkidtCbFcevPPo trec_eval_qrels_file.txt\n",
            "Processing file 1-HW4GID_ULrLq_3CDZu24RoLskqi1pj0 trec_eval_tfidf_run_file.txt\n",
            "Retrieving folder contents completed\n",
            "Building directory structure\n",
            "Building directory structure completed\n",
            "Downloading...\n",
            "From (original): https://drive.google.com/uc?id=1-6YUREBaAr5H2TNd4v66feF7-gXtRSTB\n",
            "From (redirected): https://drive.google.com/uc?id=1-6YUREBaAr5H2TNd4v66feF7-gXtRSTB&confirm=t&uuid=efa3d1d0-f279-4391-bd8f-d87b5f5735f9\n",
            "To: /content/ir-project-files/document_index.pickle.gz\n",
            "100% 29.3M/29.3M [00:00<00:00, 142MB/s] \n",
            "Downloading...\n",
            "From (original): https://drive.google.com/uc?id=1-6QY48jBNOrtppPDAV80EemnrN-f06yV\n",
            "From (redirected): https://drive.google.com/uc?id=1-6QY48jBNOrtppPDAV80EemnrN-f06yV&confirm=t&uuid=10c693fa-bc67-4edc-8318-d10eae013036\n",
            "To: /content/ir-project-files/inverted_file.pickle.gz\n",
            "100% 542M/542M [00:03<00:00, 175MB/s]\n",
            "Downloading...\n",
            "From (original): https://drive.google.com/uc?id=1-5JPLr2Ug3CTAeoYePwOui8CljX5LpV0\n",
            "From (redirected): https://drive.google.com/uc?id=1-5JPLr2Ug3CTAeoYePwOui8CljX5LpV0&confirm=t&uuid=95e838f6-3006-4a7f-b2e5-dcf2623c64fc\n",
            "To: /content/ir-project-files/lexicon.pickle.gz\n",
            "100% 40.0M/40.0M [00:00<00:00, 67.7MB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1-L0CzWJPrzmllB-3P5dE1qkV9cgyJL7G\n",
            "To: /content/ir-project-files/stats.pickle.gz\n",
            "100% 89.0/89.0 [00:00<00:00, 770kB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1-KA42mZpr_KT5E4Jl3isSThNnu2ueUg5\n",
            "To: /content/ir-project-files/trec_eval_bm25_run_file.txt\n",
            "100% 99.3k/99.3k [00:00<00:00, 90.4MB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1-Aj1Q513p315ymq_bgDkidtCbFcevPPo\n",
            "To: /content/ir-project-files/trec_eval_qrels_file.txt\n",
            "100% 219k/219k [00:00<00:00, 136MB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1-HW4GID_ULrLq_3CDZu24RoLskqi1pj0\n",
            "To: /content/ir-project-files/trec_eval_tfidf_run_file.txt\n",
            "100% 99.3k/99.3k [00:00<00:00, 168MB/s]\n",
            "Download completed\n"
          ]
        }
      ],
      "source": [
        "# Mount Google Drive to access required files\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# URL of the Google Drive repository containing the project files\n",
        "repository = \"YOUR_REPOSITORY_URL\"\n",
        "repository_name = \"ir-project-files\"\n",
        "\n",
        "# Download the specified folder from the repository\n",
        "!gdown --folder $repository\n",
        "\n",
        "# Copy the downloaded files from the repository folder to /content/\n",
        "# This ensures the files are easily accessible during execution\n",
        "for item in os.listdir(repository_name):\n",
        "  s = os.path.join(repository_name, item)\n",
        "  d = os.path.join('/content/', item)\n",
        "  if os.path.isfile(s):             # Check if the item is a file before copying\n",
        "    shutil.copy2(s, d)\n",
        "\n",
        "# Remove the downloaded repository folder to free up space\n",
        "shutil.rmtree(repository_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Noio8Q6zrwi8"
      },
      "source": [
        "# 2. Preprocessing text data\n",
        "This section defines functions for text preprocessing. Preprocessing steps include:\n",
        "- Lowercasing\n",
        "- Replacing symbols and punctuations\n",
        "- Removing stopwords\n",
        "- Stemming tokens\n",
        "\n",
        "The goal is to normalize text data for effective retrieval."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "Q1Hq10LVHcdj"
      },
      "outputs": [],
      "source": [
        "from functools import lru_cache\n",
        "import Stemmer\n",
        "nltk.download(\"stopwords\", quiet=True)\n",
        "\n",
        "# ------- Pre Initialization -------\n",
        "# Compile reusable resources and pre-load common datasets for efficiency\n",
        "# 1. Regular expression for removing unnecessary dots in acronyms\n",
        "# 2. Translation table for stripping punctuation\n",
        "# 3. Set of English stopwords for filtering irrelevant tokens\n",
        "# 4. Initialize a stemming tool for word normalization\n",
        "\n",
        "ACRONYM_REGEX = re.compile(r\"(?<!\\w)\\.(?!\\d)\")                  # Matches dots not part of decimal numbers\n",
        "PUNCTUATION_TRANS = str.maketrans(\"\", \"\", string.punctuation)   # Removes punctuation\n",
        "STOPWORDS = set(nltk.corpus.stopwords.words('english'))         # Load English stopwords\n",
        "STEMMER = Stemmer.Stemmer('english')                            # Initialize an English stemmer\n",
        "# ----------------------------------\n",
        "\n",
        "def preprocess(s):\n",
        "    \"\"\"\n",
        "    Preprocesses an input string for text analysis tasks such as indexing or querying.\n",
        "\n",
        "    Args:\n",
        "        s (str): The input string to preprocess.\n",
        "\n",
        "    Returns:\n",
        "        list[str]: A list of processed tokens.\n",
        "    \"\"\"\n",
        "\n",
        "    s = s.lower()\n",
        "    s = s.replace(\"&\", \" and \")\n",
        "    s = s.translate(str.maketrans(\"‘’´“”–-\", \"'''\\\"\\\"--\"))      # Standardize quotes and dashes for uniformity\n",
        "    s = ACRONYM_REGEX.sub(\"\", s)                                # Remove unnecessary dots in acronyms (but not decimals)\n",
        "    s = s.translate(PUNCTUATION_TRANS)                          # Remove all punctuation\n",
        "    s = \" \".join(s.split())                                     # Remove extra spaces and strip leading/trailing spaces\n",
        "\n",
        "    tokens = s.split()\n",
        "    tokens = [t for t in tokens if t not in STOPWORDS]          # Filter out stopwords\n",
        "    tokens = STEMMER.stemWords(tokens)                          # Apply stemming to normalize word forms\n",
        "    return tokens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "Bkuc9p8MHcdk"
      },
      "outputs": [],
      "source": [
        "def profile(f):\n",
        "    \"\"\"\n",
        "    A decorator to measure and print the runtime of a decorated function.\n",
        "\n",
        "    Args:\n",
        "        f (callable): The function to be profiled.\n",
        "\n",
        "    Returns:\n",
        "        callable: A wrapped version of the original function that prints its runtime after execution.\n",
        "    \"\"\"\n",
        "\n",
        "    def f_timer(*args, **kwargs):\n",
        "        \"\"\"\n",
        "        The wrapped function that measures execution time.\n",
        "\n",
        "        Args:\n",
        "            *args: Positional arguments to pass to the original function.\n",
        "            **kwargs: Keyword arguments to pass to the original function.\n",
        "\n",
        "        Returns:\n",
        "            The result of the original function.\n",
        "        \"\"\"\n",
        "\n",
        "        start = time.time()\n",
        "\n",
        "        result = f(*args, **kwargs)     # Execute the original function\n",
        "\n",
        "        end = time.time()\n",
        "        ms = (end - start) * 1000       # Calculate runtime in milliseconds\n",
        "        print(f\"{f.__name__} ({ms:.3f} ms)\")\n",
        "        return result                   # Return the result of the function\n",
        "\n",
        "    return f_timer                      # Return the wrapped function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KE16ZUvnrwi-"
      },
      "source": [
        "# 3. Building the inverted index\n",
        "We create an inverted index to store terms with their respective document IDs and term frequencies.\n",
        "The `build_index` function processes the dataset and constructs a structure that enables efficient term-based searching across documents"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "-Ugz6FpdHcdl"
      },
      "outputs": [],
      "source": [
        "@profile\n",
        "def build_index(dataset):\n",
        "    \"\"\"\n",
        "    Constructs an inverted index from a dataset.\n",
        "\n",
        "    The function processes documents to build the following components:\n",
        "    1. Lexicon: Maps terms to term IDs and tracks document frequency (DF) and term frequency (TF).\n",
        "    2. Inverted Index: Maps term IDs to lists of document IDs and term frequencies.\n",
        "    3. Document Index: A list of document IDs and their corresponding document lengths.\n",
        "    4. Index Statistics: A dictionary summarizing the index statistics.\n",
        "\n",
        "    Args:\n",
        "        dataset: The dataset to index.\n",
        "\n",
        "    Returns:\n",
        "        tuple: A tuple containing:\n",
        "            - lexicon (dict): Maps terms to [term ID, document frequency, term frequency].\n",
        "            - inverted_index (dict): Contains:\n",
        "                - 'docids' (dict): Maps term IDs to lists of document IDs.\n",
        "                - 'freqs' (dict): Maps term IDs to lists of term frequencies in the documents.\n",
        "            - document_index (list): A list of tuples (document ID, document length).\n",
        "            - stats (dict): Contains:\n",
        "                - 'num_docs': Total number of documents indexed.\n",
        "                - 'num_terms': Total number of unique terms.\n",
        "                - 'num_tokens': Total number of tokens across all documents.\n",
        "    \"\"\"\n",
        "\n",
        "    lexicon = {}                # Maps terms to [term ID, document frequency, term frequency]\n",
        "    doc_index = []              # Stores document IDs and their lengths\n",
        "    inv_d, inv_f = {}, {}       # Inverted index components: doc IDs and term frequencies\n",
        "    termid = 0                  # Counter for assigning unique term IDs\n",
        "\n",
        "    num_docs = 0                # Number of documents processed\n",
        "    total_dl = 0                # Total length of the documents (in tokens)\n",
        "\n",
        "    # Iterate over documents in the dataset\n",
        "    for docid, doc in tqdm(enumerate(dataset.docs_iter()), desc='Indexing', total=dataset.docs_count()):\n",
        "        tokens = preprocess(doc.text)               # Preprocess document text into tokens\n",
        "        token_tf = Counter(tokens)                  # Count term frequencies in the document\n",
        "\n",
        "        # Populate the lexicon and inverted index\n",
        "        for token, tf in token_tf.items():          # Assign a new term ID if the token is not in the lexicon\n",
        "            if token not in lexicon:\n",
        "                lexicon[token] = [termid, 0, 0]\n",
        "                inv_d[termid], inv_f[termid] =  [], []\n",
        "                termid += 1\n",
        "\n",
        "            token_id = lexicon[token][0]            # Get the term ID\n",
        "            inv_d[token_id].append(docid)\n",
        "            inv_f[token_id].append(tf)\n",
        "            lexicon[token][1] += 1                  # Increment document frequency for the term\n",
        "            lexicon[token][2] += tf                 # Increment total term frequency\n",
        "\n",
        "        # Update document index and statistics\n",
        "        doclen = len(tokens)\n",
        "        doc_index.append((str(doc.doc_id), doclen)) # Add document ID and length to the index\n",
        "        total_dl += doclen\n",
        "        num_docs += 1\n",
        "\n",
        "    # Build index statistics\n",
        "    stats = {\n",
        "        'num_docs': 1 + docid,                      # Total number of documents indexed\n",
        "        'num_terms': len(lexicon),                  # Total number of unique terms\n",
        "        'num_tokens': total_dl,                     # Total number of tokens across all documents\n",
        "    }\n",
        "\n",
        "    return lexicon, {'docids': inv_d, 'freqs': inv_f}, doc_index, stats"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7EPr_MAuSu5-"
      },
      "source": [
        "Operations such as tokenization, stopword removal, case normalization and stemming or lemmatization, while relatively straightforward, must be executed for every single document. Moreover, building the index requires processing each document to compute the frequency of every term. This involves updating the lexicon with the term frequency (TF) and document frequency (DF) as well as managing the inverted index by appending document IDs and term frequencies for each term in every document. This process becomes particularly resource-intensive given that the dataset contains nearly 9 million documents.\n",
        "\n",
        "Given these considerations, we aim to spare the reader of this notebook from waiting approximately 20 minutes for the index to be built. Instead, the files we downloaded earlier from Google Drive (and will now use) are the ones generated at the end of the preprocessing phase and the construction of the inverted index, along with other necessary data structures for IR."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JhOE3PCeHcdl",
        "outputId": "3397f79f-ace8-4e7e-fe6d-7b3017734383"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "All files already exist.\n"
          ]
        }
      ],
      "source": [
        "lex, inv, doc, stats = None, None, None, None               # Initialize variables for the index components\n",
        "\n",
        "files = ['lexicon.pickle.gz', 'inverted_file.pickle.gz', 'document_index.pickle.gz', 'stats.pickle.gz']\n",
        "if all(os.path.exists(file) for file in files):             # Check if all required files exist\n",
        "    print(\"All files already exist.\")\n",
        "\n",
        "    # Iterate over the list of files and their associated variable names\n",
        "    for file, var_name in zip(files, ['lex', 'inv', 'doc', 'stats']):\n",
        "        try:\n",
        "            if os.path.getsize(file) > 0:                   # Ensure the file is not empty\n",
        "                with gzip.open(file, 'rb') as f:\n",
        "                    globals()[var_name] = pickle.load(f)    # Load the file into the corresponding variable\n",
        "            else:\n",
        "                print(f\"Warning: {file} is empty.\")\n",
        "        except EOFError:\n",
        "            # If the file is corrupted or incomplete, rebuild the index\n",
        "            print(f\"Error: {file} is corrupted or incomplete. Rebuilding the index.\")\n",
        "            lex, inv, doc, stats = build_index(dataset)\n",
        "            break\n",
        "else:\n",
        "    # If any of the files do not exist, rebuild the index\n",
        "    lex, inv, doc, stats = build_index(dataset)\n",
        "\n",
        "    # Save the rebuilt index components back into the respective files\n",
        "    for data, file in zip([lex, inv, doc, stats], files):\n",
        "      with gzip.open(file, 'wb') as f:\n",
        "        print(f\"Saving {file}...\")\n",
        "        pickle.dump(data, f)                                # Serialize and save the data\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "4BRtzyxlHcdm"
      },
      "outputs": [],
      "source": [
        "class InvertedIndex:\n",
        "    \"\"\"\n",
        "    A simple inverted index class. Stores term-document mappings for fast retrieval.\n",
        "\n",
        "    Attributes:\n",
        "        lexicon (dict): Maps a token to [termID, docFreq, totalTermFreq].\n",
        "        inv (dict): Contains 'docids' and 'freqs' lists, indexed by termID.\n",
        "        doc (list): Each element is (doc_id, doc_length).\n",
        "        stat (dict): Index statistics (e.g., num_docs, num_terms, num_tokens).\n",
        "\n",
        "    Methods:\n",
        "        num_docs() -> int\n",
        "            Returns the total number of indexed documents.\n",
        "        get_posting(termid: int) -> PostingListIterator\n",
        "            Returns a posting list iterator for the given termID.\n",
        "        get_termids(tokens: list[str]) -> list[int]\n",
        "            Converts tokens to termIDs if found in the lexicon.\n",
        "        get_postings(termids: list[int]) -> list[PostingListIterator]\n",
        "            Returns posting list iterators for each termID.\n",
        "    \"\"\"\n",
        "\n",
        "    class PostingListIterator:\n",
        "        \"\"\"\n",
        "        (Inner class) Iterates over the posting list for a single termID.\n",
        "\n",
        "        Attributes:\n",
        "            docids (list[int]): Document IDs containing this term.\n",
        "            freqs (list[int]): Term frequencies in the corresponding docID.\n",
        "            pos (int): Current index in the posting list.\n",
        "            doc (list): Reference to the main document index.\n",
        "\n",
        "        Methods:\n",
        "            docid() -> int or math.inf\n",
        "                Returns the current docID or math.inf if finished.\n",
        "            score() -> float or math.inf\n",
        "                Returns freq / doc_length or math.inf if finished.\n",
        "            next(target: int = None) -> None\n",
        "                Moves forward or jumps to target docID if specified.\n",
        "            is_end_list() -> bool\n",
        "                Checks if the iterator has reached the end.\n",
        "            len() -> int\n",
        "                Returns the total number of docIDs for this term.\n",
        "        \"\"\"\n",
        "\n",
        "        def __init__(self, docids, freqs, doc):\n",
        "            \"\"\"\n",
        "            Initialize the iterator with document IDs, frequencies, and a reference to the document index.\n",
        "            \"\"\"\n",
        "            self.docids = docids            # List of document IDs where the term appears\n",
        "            self.freqs = freqs              # List of term frequencies corresponding to each document ID\n",
        "            self.pos = 0                    # Start position in the posting list\n",
        "            self.doc = doc                  # Reference to the main document index\n",
        "\n",
        "        def docid(self):\n",
        "            \"\"\"\n",
        "            Returns the current document ID or math.inf if the end of the list is reached.\n",
        "            \"\"\"\n",
        "            if self.is_end_list():\n",
        "                return math.inf\n",
        "            return self.docids[self.pos]\n",
        "\n",
        "        def score(self):\n",
        "            \"\"\"\n",
        "            Computes the term frequency normalized by the document length for the current position.\n",
        "            Returns math.inf if the end of the list is reached.\n",
        "            \"\"\"\n",
        "            if self.is_end_list():\n",
        "                return math.inf\n",
        "            return self.freqs[self.pos]/self.doc[self.docid()][1]\n",
        "\n",
        "        def next(self, target = None):\n",
        "            \"\"\"\n",
        "            Advances to the next position in the posting list or jumps to the target document ID.\n",
        "            \"\"\"\n",
        "            if not target:                              # If no target is specified, move to the next position\n",
        "                if not self.is_end_list():\n",
        "                    self.pos += 1\n",
        "            else:\n",
        "                if target > self.docid():               # If a target is specified, jump to its position if it exists\n",
        "                    try:\n",
        "                        self.pos = self.docids.index(target, self.pos)\n",
        "                    except ValueError:\n",
        "                        self.pos = len(self.docids)     # Move to the end if the target is not found\n",
        "\n",
        "        def is_end_list(self):\n",
        "            \"\"\"\n",
        "            Checks if the iterator has reached the end of the posting list.\n",
        "            \"\"\"\n",
        "            return self.pos == len(self.docids)\n",
        "\n",
        "\n",
        "        def len(self):\n",
        "            \"\"\"\n",
        "            Returns the total number of document IDs in the posting list.\n",
        "            \"\"\"\n",
        "            return len(self.docids)\n",
        "\n",
        "\n",
        "    def __init__(self, lex, inv, doc, stats):\n",
        "        \"\"\"\n",
        "        Initialize the inverted index with its components: lexicon, inverted file, document index, and stats.\n",
        "        \"\"\"\n",
        "        self.lexicon = lex          # Lexicon mapping tokens to [termID, docFreq, totalTermFreq]\n",
        "        self.inv = inv              # Inverted index with 'docids' and 'freqs'\n",
        "        self.doc = doc              # List of documents with IDs and lengths\n",
        "        self.stat = stats           # Index statistics (e.g., number of documents, terms, tokens)\n",
        "\n",
        "    def num_docs(self):\n",
        "        \"\"\"\n",
        "        Returns the total number of indexed documents.\n",
        "        \"\"\"\n",
        "        return self.stats['num_docs']\n",
        "\n",
        "    def get_posting(self, termid):\n",
        "        \"\"\"\n",
        "        Returns a PostingListIterator for the given term ID.\n",
        "        \"\"\"\n",
        "        return InvertedIndex.PostingListIterator(self.inv['docids'][termid], self.inv['freqs'][termid], self.doc)\n",
        "\n",
        "    def get_termids(self, tokens):\n",
        "        \"\"\"\n",
        "        Converts a list of tokens to their corresponding term IDs using the lexicon.\n",
        "        \"\"\"\n",
        "        return [self.lexicon[token][0] for token in tokens if token in self.lexicon]\n",
        "\n",
        "    def get_postings(self, termids):\n",
        "        \"\"\"\n",
        "        Returns a list of PostingListIterators for the given term IDs.\n",
        "        \"\"\"\n",
        "        return [self.get_posting(termid) for termid in termids]\n",
        "\n",
        "inv_index = InvertedIndex(lex, inv, doc, stats)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XA-LFkjhHcdm"
      },
      "source": [
        "# 4. Query processing\n",
        "This section implements the Query Processing task, aiming to rank documents by relevance to a given query using the BM25 and TF-IDF scoring functions with two different approaches:\n",
        "- **DAAT (Document-at-a-Time)**: Processes documents sequentially, computing scores for all terms in a document before moving to the next document.\n",
        "- **TAAT (Term-at-a-Time)**: Processes terms sequentially, scoring all documents for a given term before moving to the next term."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "Edu7ZtPfHcdn"
      },
      "outputs": [],
      "source": [
        "trec_dl_2020 = ir_datasets.load(\"msmarco-passage/trec-dl-2020\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "ywDp7MMVHcdn"
      },
      "outputs": [],
      "source": [
        "class TopQueue:\n",
        "    \"\"\"\n",
        "    A simple top-k priority queue to maintain the top-scoring items.\n",
        "\n",
        "    This class uses a min-heap to efficiently store and retrieve the top-k\n",
        "    items based on their scores. Items are tuples of (score, docid).\n",
        "\n",
        "    Attributes:\n",
        "        queue (list[tuple[float, int]]): The priority queue storing (score, docid) pairs.\n",
        "        k (int): The maximum number of items to maintain in the queue.\n",
        "        threshold (float): The minimum score required for an item to enter the queue.\n",
        "\n",
        "    Methods:\n",
        "        size() -> int:\n",
        "            Returns the current number of items in the queue.\n",
        "        would_enter(score: float) -> bool:\n",
        "            Checks if a given score exceeds the threshold and could enter the queue.\n",
        "        clear(new_threshold: float = None) -> None:\n",
        "            Clears the queue and optionally sets a new threshold.\n",
        "        insert(docid: int, score: float) -> bool:\n",
        "            Attempts to insert an item into the queue. Updates the threshold if needed.\n",
        "        __repr__() -> str:\n",
        "            Returns a string representation of the queue.\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, k=10, threshold=0.0):\n",
        "        \"\"\"\n",
        "        Initializes the TopQueue with a maximum size and an optional threshold.\n",
        "        \"\"\"\n",
        "        self.queue = []                     # Initialize an empty priority queue (min-heap)\n",
        "        self.k = k                          # Maximum number of items to store\n",
        "        self.threshold = threshold          # Initial score threshold\n",
        "\n",
        "    def size(self):\n",
        "        \"\"\"\n",
        "        Returns the current number of items in the queue.\n",
        "        \"\"\"\n",
        "        return len(self.queue)\n",
        "\n",
        "    def would_enter(self, score):\n",
        "        \"\"\"\n",
        "        Checks if a given score exceeds the current threshold and could enter the queue.\n",
        "        \"\"\"\n",
        "        return score > self.threshold\n",
        "\n",
        "    def clear(self, new_threshold=None):\n",
        "        \"\"\"\n",
        "        Clears all items from the queue and optionally sets a new threshold.\n",
        "        \"\"\"\n",
        "        self.queue = []                     # Empty the queue\n",
        "        if new_threshold is not None:\n",
        "            self.threshold = new_threshold  # Update the threshold if provided\n",
        "\n",
        "    def __repr__(self):\n",
        "        \"\"\"\n",
        "        Returns a string representation of the queue.\n",
        "        \"\"\"\n",
        "        return f'<{self.size()} items, th={self.threshold} {self.queue}>'\n",
        "\n",
        "    def insert(self, docid, score):\n",
        "        \"\"\"\n",
        "        Attempts to insert an item into the queue. Maintains the top-k items by score.\n",
        "        \"\"\"\n",
        "        if score > self.threshold:\n",
        "            if self.size() >= self.k:                                   # If the queue is full\n",
        "                heapq.heapreplace(self.queue, (score, docid))           # Replace the smallest item\n",
        "            else:\n",
        "                heapq.heappush(self.queue, (score, docid))              # Add the item to the queue\n",
        "\n",
        "            if self.size() >= self.k:                                   # Update the threshold if the queue is full\n",
        "                self.threshold = max(self.threshold, self.queue[0][0])  # The lowest score becomes the threshold\n",
        "            return True\n",
        "        return False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7zjd0AxOrwjA"
      },
      "source": [
        "## 4.1. BM25"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a9okn1a6Su6A"
      },
      "source": [
        "This BM25 implementation is based on the classic formulation but includes several optimizations to align with modern variants, such as those proposed by the PyTerrier library. <br>\n",
        "Specifically, a base-2 logarithmic transformation is used for the IDF factor calculation, aligning with common practices in Information Retrieval benchmarks. The document length is normalized against the average collection length (`avg_dl`), and the parameters `k1`​ and `b` control the saturation of term frequency and the relative weight of length normalization, respectively. <br>\n",
        "This variant is designed to enable direct comparison with other implementations in benchmarks, balancing computational efficiency and accuracy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CpiyXRWDSu6A"
      },
      "source": [
        "### Formula of BM25\n",
        "\n",
        "$\\text{BM25} = \\log_2\\left(1 + \\frac{N - \\text{df} + 0.5}{\\text{df} + 0.5}\\right)$\n",
        "$\\cdot \\frac{(k_1 + 1) \\cdot \\text{tf}}{K + \\text{tf}}$\n",
        "\n",
        "Where:\n",
        "\n",
        "$K = k_1 \\cdot \\left((1 - b) + b \\cdot \\frac{\\text{dl}}{\\text{avg\\_dl}}\\right)$\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "l6naDw7fHcdn"
      },
      "outputs": [],
      "source": [
        "LOG_E_OF_2 = math.log(2)        # Natural logarithm of 2 for base conversion\n",
        "LOG_2_OF_E = 1 / LOG_E_OF_2     # Conversion factor for log base-e to base-2\n",
        "\n",
        "# Compute average document length and total number of documents from the index\n",
        "avg_dl = inv_index.stat['num_tokens'] / inv_index.stat['num_docs']\n",
        "N = inv_index.stat['num_docs']\n",
        "\n",
        "def bm25(tf, df, dl, k1=1.2, b=0.75):\n",
        "    \"\"\"\n",
        "    Compute the BM25 relevance score for a term in a document.\n",
        "\n",
        "    Args:\n",
        "        tf (int):   Term frequency, the count of the term in the document.\n",
        "        df (int):   Document frequency, the number of documents containing the term.\n",
        "        dl (float): Document length, the number of tokens in the document.\n",
        "        k1 (float): Parameter controlling term frequency saturation.\n",
        "        b (float):  Parameter controlling document length normalization.\n",
        "\n",
        "    Returns:\n",
        "        float: The BM25 score for the term in the document with respect to the query.\n",
        "    \"\"\"\n",
        "    idf = math.log(1 + (N - df + 0.5) / (df + 0.5)) * LOG_2_OF_E                # IDF weighting\n",
        "    K = k1 * ((1 - b) + b * (dl / avg_dl))                                      # Document length adjustment\n",
        "    term_frequency_component = ((k1 + 1) * tf) / (K + tf)                       # TF component\n",
        "    return idf * term_frequency_component"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brsGzbDNrwjA"
      },
      "source": [
        "### 4.1.1 DAAT with BM25"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "K3NbYKX2Hcdn"
      },
      "outputs": [],
      "source": [
        "# Precompute document lengths\n",
        "doc_lengths = defaultdict(int)\n",
        "for docid, doc_len in inv_index.doc:\n",
        "    doc_lengths[docid] = doc_len\n",
        "\n",
        "def min_docid(postings):\n",
        "    \"\"\"\n",
        "    Find the smallest document ID among active posting list iterators.\n",
        "\n",
        "    Args:\n",
        "        postings (list[PostingListIterator]): Posting list iterators.\n",
        "\n",
        "    Returns:\n",
        "        int: The smallest document ID or math.inf if all lists are exhausted.\n",
        "    \"\"\"\n",
        "\n",
        "    min_docid = math.inf\n",
        "    for p in postings:\n",
        "        if not p.is_end_list():     # Skip completed lists\n",
        "            min_docid = min(p.docid(), min_docid)\n",
        "    return min_docid\n",
        "\n",
        "def daat_bm25(postings, k=10):\n",
        "    \"\"\"\n",
        "    Perform Document-At-A-Time (DAAT) retrieval with BM25 scoring.\n",
        "\n",
        "    Args:\n",
        "        postings (list[PostingListIterator]): Posting lists for terms.\n",
        "        k (int): Number of top results to retrieve.\n",
        "\n",
        "    Returns:\n",
        "        list[tuple[int, float]]: Top-k (docid, score) pairs sorted by score.\n",
        "    \"\"\"\n",
        "\n",
        "    top = TopQueue(k)                               # Initialize top-k priority queue\n",
        "    current_docid = min_docid(postings)             # Start with the smallest document ID\n",
        "\n",
        "    while current_docid != math.inf:                # Process documents until all posting lists are exhausted\n",
        "        score = 0\n",
        "        next_docid = math.inf\n",
        "\n",
        "        for posting in postings:\n",
        "            if posting.docid() == current_docid:    # Check if the term is in the current doc\n",
        "                tf = posting.freqs[posting.pos]\n",
        "                df = posting.len()\n",
        "                dl = doc_lengths[current_docid]\n",
        "\n",
        "                score += bm25(tf, df, dl)\n",
        "\n",
        "                posting.next()                      # Move to the next term occurrence\n",
        "\n",
        "            if not posting.is_end_list():           # Update the smallest doc ID for next iteration\n",
        "                next_docid = min(next_docid, posting.docid())\n",
        "\n",
        "        top.insert(current_docid, score)            # Add the current doc to the top-k queue\n",
        "        current_docid = next_docid                  # Move to the next document\n",
        "\n",
        "    return sorted(top.queue, reverse=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E9Eub_uGHcdo"
      },
      "source": [
        "### 4.1.2 TAAT with BM25"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "P85FZGrxHcdo"
      },
      "outputs": [],
      "source": [
        "def taat_bm25(postings, k=10):\n",
        "    \"\"\"\n",
        "    Perform Term-At-A-Time (TAAT) retrieval with BM25 scoring.\n",
        "\n",
        "    Args:\n",
        "        postings (list[PostingListIterator]): A list of posting list iterators, one for each query term.\n",
        "        k (int): The maximum number of top documents to retrieve. Default is 10.\n",
        "\n",
        "    Returns:\n",
        "        list[tuple[int, float]]: A sorted list of (docid, score) tuples, ordered by score in descending order.\n",
        "    \"\"\"\n",
        "    A = defaultdict(float)                      # Accumulator for document scores\n",
        "\n",
        "    # Process one term's posting list at a time\n",
        "    for posting in postings:\n",
        "        current_docid = posting.docid()\n",
        "        df = posting.len()                      # Document frequency for the current term\n",
        "\n",
        "        while current_docid != math.inf:\n",
        "            tf = posting.freqs[posting.pos]     # Term frequency in the current document\n",
        "            dl = doc_lengths[current_docid]     # Length of the current document\n",
        "\n",
        "            score = bm25(tf, df, dl)            # Compute BM25 score for the term-document pair\n",
        "            A[current_docid] += score\n",
        "\n",
        "            posting.next()\n",
        "            current_docid = posting.docid()\n",
        "\n",
        "    top = TopQueue(k)\n",
        "\n",
        "    for docid, score in A.items():              # Insert all documents and their scores into the top-k queue\n",
        "        top.insert(docid, score)\n",
        "\n",
        "    return sorted(top.queue, reverse=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "41JK5qIJrwjA"
      },
      "source": [
        "## 4.2 TF-IDF"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5mK-UgMKSu6B"
      },
      "source": [
        "The TF-IDF we use is not the standard formula, but a variant of it. To define it, we were inspired by the version used in the [PyTerrier library](https://github.com/terrier-org/pyterrier), where both term frequency saturation and document length are considered. This also allows for a direct comparison with that library in Information Retrieval benchmarks.\n",
        "\n",
        "$\\text{TF-IDF} = \\text{TF}_{\\text{Robertson}} \\cdot \\text{IDF}$\n",
        "\n",
        "Where:\n",
        "\n",
        "$\\text{TF}_{\\text{Robertson}} = \\frac{k_1 \\cdot \\text{tf}}{\\text{tf} + k_1 \\cdot \\left((1 - b) + b \\cdot \\frac{\\text{dl}}{\\text{avg\\_dl}}\\right)}$\n",
        "\n",
        "$\\text{IDF} = \\log_2\\left(\\frac{N}{\\text{df}} + 1\\right)$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "IQ_8_wSbrwjA"
      },
      "outputs": [],
      "source": [
        "def tfidf_score(tf, df, dl, k1 = 1.2, b = 0.75):\n",
        "    \"\"\"\n",
        "    Compute the TF-IDF score using a normalized term frequency formulation.\n",
        "\n",
        "    Args:\n",
        "        tf (int): Term frequency in the document.\n",
        "        df (int): Document frequency, the number of documents containing the term.\n",
        "        dl (float): Document length, the total number of tokens in the document.\n",
        "        k1 (float): Term frequency saturation parameter.\n",
        "        b (float): Length normalization parameter.\n",
        "\n",
        "    Returns:\n",
        "        float: The TF-IDF score for the term in the document with respect to the query.\n",
        "    \"\"\"\n",
        "    # Compute normalized term frequency\n",
        "    tf_robertson = k1 * tf / (tf + (k1 * ((1 - b) + ((b * dl) / avg_dl))))\n",
        "    # Compute inverse document frequency (IDF) with base-2 logarithm\n",
        "    idf = math.log((N / df) + 1) * LOG_2_OF_E\n",
        "\n",
        "    return tf_robertson * idf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_epDLSRwrwjA"
      },
      "source": [
        "### 4.2.1 DAAT with TF-IDF"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "ebv3gm8NrwjA"
      },
      "outputs": [],
      "source": [
        "def daat_tfidf(postings, k=10):\n",
        "    \"\"\"\n",
        "    Perform Document-At-A-Time (DAAT) retrieval using TF-IDF scoring.\n",
        "\n",
        "    Args:\n",
        "        postings (list[PostingListIterator]): A list of posting list iterators, one for each query term.\n",
        "        k (int): The maximum number of top documents to retrieve. Default is 10.\n",
        "\n",
        "    Returns:\n",
        "        list[tuple[int, float]]: A sorted list of (docid, score) tuples, ordered by score in descending order.\n",
        "    \"\"\"\n",
        "\n",
        "    top = TopQueue(k)                               # Initialize a priority queue for the top-k results\n",
        "    current_docid = min_docid(postings)             # Start with the smallest document ID across postings\n",
        "\n",
        "    while current_docid != math.inf:                # Loop until all documents are processed\n",
        "        score = 0\n",
        "        next_docid = math.inf\n",
        "\n",
        "        for posting in postings:\n",
        "            if posting.docid() == current_docid:    # Check if the term appears in the current document\n",
        "                tf = posting.freqs[posting.pos]\n",
        "                df = posting.len()\n",
        "                dl = doc_lengths[current_docid]\n",
        "\n",
        "                score += tfidf_score(tf, df, dl)    # Accumulate the TF-IDF score for this document\n",
        "\n",
        "                posting.next()\n",
        "\n",
        "            if not posting.is_end_list():           # Update the next smallest document ID\n",
        "                next_docid = min(next_docid, posting.docid())\n",
        "\n",
        "        top.insert(current_docid, score)            # Insert the document and its score into the top-k queue\n",
        "        current_docid = next_docid                  # Move to the next document to be scored\n",
        "\n",
        "    return sorted(top.queue, reverse=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kulAlTcXrwjB"
      },
      "source": [
        "### 4.2.2 TAAT with TF-IDF"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "v-9eM3z2rwjB"
      },
      "outputs": [],
      "source": [
        "def taat_tfidf(postings, k=10):\n",
        "    \"\"\"\n",
        "    Perform Term-At-A-Time (TAAT) retrieval using TF-IDF scoring.\n",
        "\n",
        "    Args:\n",
        "        postings (list[PostingListIterator]): A list of posting list iterators, one for each query term.\n",
        "        k (int): The maximum number of top documents to retrieve.\n",
        "\n",
        "    Returns:\n",
        "        list[tuple[int, float]]: A sorted list of (docid, score) tuples, ordered by score in descending order.\n",
        "    \"\"\"\n",
        "    A = defaultdict(float)                      # Accumulator for document scores\n",
        "\n",
        "    # Process one term's posting list at a time\n",
        "    for posting in postings:\n",
        "        current_docid = posting.docid()\n",
        "\n",
        "        df = posting.len()\n",
        "\n",
        "        while current_docid != math.inf:\n",
        "            tf = posting.freqs[posting.pos]\n",
        "            dl = doc_lengths[current_docid]\n",
        "\n",
        "            score = tfidf_score(tf, df, dl)      # Compute TF-IDF score\n",
        "            A[current_docid] += score\n",
        "\n",
        "            posting.next()\n",
        "            current_docid = posting.docid()\n",
        "\n",
        "    top = TopQueue(k)\n",
        "\n",
        "    for docid, score in A.items():              # Insert all documents and their scores into the top-k queue\n",
        "        top.insert(docid, score)\n",
        "\n",
        "    return sorted(top.queue, reverse=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "isLrTRUIrwjB"
      },
      "source": [
        "## 4.3 Results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "EmKXPJvBHcdo"
      },
      "outputs": [],
      "source": [
        "@profile\n",
        "def query_processing(queries_iter, fn):\n",
        "    \"\"\"\n",
        "    Process a list of queries using a specified scoring function.\n",
        "\n",
        "    Args:\n",
        "        queries_iter (iterable): An iterable of query objects.\n",
        "        fn (callable): A scoring function that takes a list of posting list iterators\n",
        "            and returns a list of (docid, score) tuples.\n",
        "\n",
        "    Returns:\n",
        "        list[dict]: A list of results, each containing:\n",
        "            - `query_id` (int): The ID of the processed query.\n",
        "            - `scores` (list[tuple[int, float]]): The list of (docid, score) tuples for the query.\n",
        "    \"\"\"\n",
        "\n",
        "    res = []                                # Store the results for each query\n",
        "\n",
        "    for q in queries_iter:\n",
        "        query = preprocess(q.text)                  # Preprocess the query text\n",
        "        termids = inv_index.get_termids(query)      # Map query tokens to term IDs\n",
        "        postings = inv_index.get_postings(termids)  # Retrieve posting lists for the term IDs\n",
        "\n",
        "        # Compute scores using the provided scoring function and store the result\n",
        "        res.append({'query_id': q.query_id, 'scores': fn(postings)})\n",
        "\n",
        "    return res"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "om92ICN-Hcdo",
        "outputId": "a228aaa1-2dd9-4676-eb28-45060f39473f"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[INFO] Please confirm you agree to the MSMARCO data usage agreement found at <http://www.msmarco.org/dataset.aspx>\n",
            "[INFO] [starting] https://msmarco.z22.web.core.windows.net/msmarcoranking/msmarco-test2020-queries.tsv.gz\n",
            "[INFO] [finished] https://msmarco.z22.web.core.windows.net/msmarcoranking/msmarco-test2020-queries.tsv.gz: [00:00] [4.13kB] [30.9MB/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "query_processing (318644.317 ms)\n",
            "[{'query_id': '1030303', 'scores': [(68.04124339861062, 8726436), (60.1427866714432, 8726435), (60.1427866714432, 8726433), (60.1427866714432, 8726429), (56.26487434885108, 8726437), (56.26487434885108, 8726434), (56.26487434885108, 8726430), (56.26487434885108, 7156982), (36.8093899826997, 1305521), (36.8093899826997, 1305520)]}, {'query_id': '1037496', 'scores': [(49.78270676861767, 7766587), (49.78270676861767, 7766585), (49.78270676861767, 5927420), (49.78270676861767, 4760912), (46.08250624801771, 4905511), (46.08250624801771, 4760914), (46.08250624801771, 4174377), (46.08250624801771, 3725937), (34.3057350592058, 4174376), (33.526059262405674, 4174378)]}, {'query_id': '1043135', 'scores': [(67.58094550401972, 8696961), (65.84778521905021, 650642), (64.07233051343995, 3378240), (63.811499082452244, 8696958), (62.87382325532231, 4994428), (61.23387105030339, 4355523), (61.23387105030339, 2514458), (60.077594538221234, 650641), (57.61450176736487, 2664995), (56.49291203767402, 4721484)]}, {'query_id': '1045109', 'scores': [(62.65581464672154, 8838896), (55.20688108779858, 8838890), (52.88217275039405, 8838893), (52.59407248914457, 8838892), (51.588943666145575, 8838891), (49.03746280776349, 8838899), (49.03746280776349, 8838898), (49.03746280776349, 8838897), (49.03746280776349, 8838895), (38.51262152953826, 7620329)]}, {'query_id': '1049519', 'scores': [(51.868481761168326, 7105), (50.76958755275562, 5355123), (46.32118326232374, 1316185), (45.971381551470046, 7106), (45.20014989004244, 7668125), (41.307972299785845, 5355121), (40.751745599610565, 6080565), (40.26732811546444, 5355126), (39.890437244812055, 6950445), (39.890437244812055, 6580258)]}, {'query_id': '1051399', 'scores': [(54.19868414964971, 4426188), (54.19868414964971, 69818), (54.19868414964971, 69813), (53.965490707744955, 2845499), (53.171797439158496, 7848674), (53.108276110252035, 3773237), (52.55272840666184, 8712317), (52.193951693351394, 2866972), (51.76370019693758, 3446937), (51.53270911799906, 3943228)]}, {'query_id': '1056416', 'scores': [(79.61916811336067, 8739209), (75.11227933549185, 2308626), (68.87502469490411, 8757245), (68.87502469490411, 2059786), (65.11806034305253, 3790242), (64.89493842990825, 895432), (63.01310460438273, 7321435), (62.78998269123845, 2348501), (62.78998269123845, 1367298), (59.07809351037385, 2348504)]}, {'query_id': '1064670', 'scores': [(60.32750960955309, 8680901), (58.767713457504655, 8680903), (58.085711433226464, 8680905), (56.959861539866935, 8680897), (54.612497156036284, 8680904), (52.37069897970967, 8680902), (45.323465075433155, 7734392), (43.08166689910654, 3807086), (41.95581700574701, 8337453), (41.95581700574701, 6878684)]}, {'query_id': '1065636', 'scores': [(58.78932722598091, 4371749), (57.178669194806446, 1062134), (57.02821144543303, 8490752), (49.19777543437664, 7411479), (48.34522226593835, 436583), (48.074107885396785, 3822992), (48.074107885396785, 597867), (47.806681760757506, 6307138), (47.32523509989382, 7411495), (47.18849018695526, 444919)]}, {'query_id': '1071750', 'scores': [(96.42909288222162, 3928913), (93.4766731950148, 8698268), (89.47034086348212, 8698276), (87.24605139920457, 8698274), (87.24605139920457, 8698269), (87.24605139920457, 3575934), (84.62555403932791, 8378390), (74.19949783794277, 8378386), (71.57900047806609, 8378389), (71.57900047806609, 8378385)]}, {'query_id': '1103153', 'scores': [(46.94883239928314, 8226438), (46.412659612159885, 8226436), (45.694343174868166, 8226445), (45.64339609448325, 7570096), (44.9960865467194, 6113506), (43.690650241919506, 8226444), (43.690650241919506, 8226440), (43.690650241919506, 8226439), (42.329596243837386, 8226443), (42.329596243837386, 8226441)]}, {'query_id': '1103791', 'scores': [(31.666238533045593, 8384424), (31.17049503301221, 8384416), (30.886013979015406, 4034845), (30.697285328434887, 8220282), (30.534630879918826, 4034852), (30.2830101983892, 1512326), (30.270988726409414, 8384417), (30.270988726409414, 8146212), (29.976348721672593, 8220278), (29.976348721672593, 8214165)]}, {'query_id': '1104501', 'scores': [(65.89412729708552, 6871582), (65.41840449297159, 6067096), (65.39582284977347, 5286080), (64.61878497960438, 2204057), (64.54957712134694, 2204054), (64.43256917565267, 2090898), (64.31747924474085, 5594847), (64.19704858360912, 7431269), (64.15606231274846, 2283306), (64.12101683648413, 3604439)]}, {'query_id': '1105792', 'scores': [(39.37059315121416, 8201616), (39.37059315121416, 8201611), (38.999172461108365, 8201618), (38.999172461108365, 8201612), (38.45499796165104, 8201613), (37.581020735249886, 8201615), (37.581020735249886, 8201610), (37.581020735249886, 6977017), (35.94706331197815, 8201614), (35.94706331197815, 7102720)]}, {'query_id': '1105860', 'scores': [(51.80051982448995, 3593906), (51.80051982448995, 3593905), (51.80051982448995, 3593904), (51.34894599516154, 8200940), (51.2784025316851, 8282567), (51.2784025316851, 8282566), (51.2784025316851, 8282565), (50.90628181279334, 7237676), (50.87358981321845, 3593907), (50.65799606338433, 4866505)]}, {'query_id': '1106928', 'scores': [(59.20863218150503, 4285175), (57.68234298179986, 2168463), (55.4006948609547, 5186636), (54.77346628597604, 7744803), (54.20077932621291, 8190627), (54.20077932621291, 8190624), (53.27899157588621, 2866124), (53.155024758113754, 8190620), (52.82548564081675, 1392371), (51.51175578482002, 6949062)]}, {'query_id': '1106979', 'scores': [(60.53563858569951, 8190140), (60.53563858569951, 8190137), (60.35317454410322, 7674419), (55.651224895562486, 7674423), (54.4033093271833, 5383176), (54.4033093271833, 4219185), (54.4033093271833, 1589841), (52.707424067591056, 8190135), (49.86026336047267, 8190134), (49.162371301808044, 348057)]}, {'query_id': '1107315', 'scores': [(39.54376615657153, 4598318), (37.51389793839698, 8187302), (36.022464891991085, 8817722), (36.022464891991085, 2696421), (36.022464891991085, 2696417), (36.022464891991085, 2696416), (35.22311638066542, 2687026), (35.22311638066542, 2687024), (35.22311638066542, 1708353), (34.15680819295444, 6437724)]}, {'query_id': '1107440', 'scores': [(61.89699856109981, 8186231), (58.33778121442552, 8186223), (45.6775153791329, 789989), (43.80313314161787, 1409567), (38.523388107855446, 3971063), (37.647486625284195, 1879424), (37.17878681008612, 8186227), (37.17878681008612, 5177729), (37.17878681008612, 5177724), (35.989959445628465, 2888123)]}, {'query_id': '1108450', 'scores': [(56.71887953761221, 1629571), (52.41776940719191, 4400204), (32.300004292406435, 1629577), (31.777632255332467, 6285212), (31.777632255332467, 6285210), (31.33422343316504, 8177145), (31.33422343316504, 4747082), (31.33422343316504, 4400207), (31.33422343316504, 4400205), (31.33422343316504, 1629573)]}, {'query_id': '1108466', 'scores': [(59.30142271208983, 8177012), (55.67270254789683, 4944265), (55.67270254789683, 3267306), (55.67270254789683, 753377), (55.199872139350035, 2702004), (54.82411557645084, 841054), (54.51524769260299, 5380573), (54.51524769260299, 3482588), (54.51524769260299, 3267303), (54.40273520904941, 2002688)]}, {'query_id': '1108473', 'scores': [(71.8535841426751, 8176948), (68.837321617921, 1188068), (67.88159874897809, 1188069), (64.05070212441933, 8176949), (56.883210440504, 1970473), (56.391214766044264, 3978700), (55.26259023758357, 8176947), (54.75703926520205, 1188063), (53.55061486545402, 5562098), (53.50519901171889, 357762)]}, {'query_id': '1108651', 'scores': [(52.042835175291344, 6348741), (51.707125565958556, 8175412), (48.84626280639219, 1371583), (47.09379244246787, 5027435), (43.784108385907516, 6985088), (43.784108385907516, 5997928), (43.29405003861879, 6663873), (42.9854927883285, 1325612), (42.88347701771074, 6985085), (42.88347701771074, 4617850)]}, {'query_id': '1108729', 'scores': [(63.10547382702718, 3405694), (57.22452254923036, 7789132), (53.69516866941264, 5449883), (48.865078065808376, 3405688), (46.03079622421798, 8174842), (44.74012911893431, 8322380), (44.46628101333716, 334958), (44.164593352223406, 7993178), (43.79687803700468, 1820798), (43.73131067345461, 5623921)]}, {'query_id': '1109699', 'scores': [(32.67479266338443, 2769074), (32.65745417997212, 4872045), (32.56902159014303, 8394502), (32.45251516227297, 7722707), (32.45251516227297, 7722706), (32.45251516227297, 7471054), (32.45251516227297, 2341119), (32.45251516227297, 737337), (32.30683047648765, 7744060), (32.30683047648765, 178437)]}, {'query_id': '1109707', 'scores': [(63.99305622763602, 1911564), (62.17738366690502, 1719374), (61.976129020952484, 5947139), (61.95931813768323, 4122615), (61.09866103592489, 771002), (60.89949286673625, 4122618), (60.87968502011701, 1719371), (60.53769115031774, 5740304), (60.192034498712474, 4285843), (59.92002711015108, 4099609)]}, {'query_id': '1109850', 'scores': [(60.34972932491009, 8164028), (58.28235902800127, 8164035), (58.28235902800127, 8164030), (56.74119317420177, 8164033), (46.99295859198112, 3633621), (46.99295859198112, 3633617), (42.70642938809392, 8164032), (40.5782020099497, 8164036), (33.55505166207379, 8164029), (28.039054851875697, 5382247)]}, {'query_id': '1110678', 'scores': [(47.78458294762738, 7139329), (45.84152116584514, 8155914), (45.84152116584514, 6632692), (45.84152116584514, 2342590), (45.84152116584514, 2342589), (44.97464403661556, 8155909), (44.97464403661556, 5719597), (42.62799591135909, 2714688), (42.62799591135909, 2126363), (42.62799591135909, 2096160)]}, {'query_id': '1112142', 'scores': [(81.1818694630267, 2542782), (80.2780492424212, 3651292), (80.2780492424212, 3651291), (80.2780492424212, 3651290), (79.8570649461349, 7866090), (72.65079473401737, 2238202), (69.50557890927026, 1946438), (68.23903268901505, 5670185), (68.23903268901505, 4967524), (68.23903268901505, 3974360)]}, {'query_id': '1113042', 'scores': [(50.769852428627246, 7072957), (50.36054095803586, 2410652), (49.2616863479491, 355186), (48.33019169852579, 2499270), (47.863811684552275, 3521102), (46.99188383234091, 7834150), (46.99188383234091, 4931496), (46.99188383234091, 4243566), (46.454216943749486, 2326525), (46.454216943749486, 2326519)]}, {'query_id': '1113256', 'scores': [(92.31619375511241, 7561115), (91.67121386472638, 8130535), (91.67121386472638, 7561120), (89.3871919777977, 7561117), (87.40058143324417, 7561116), (86.7519549223406, 7561114), (86.05712136539144, 8130530), (86.05712136539144, 7561113), (83.42553093045187, 7561111), (76.74533982637449, 8130528)]}, {'query_id': '1114166', 'scores': [(43.684903129639494, 6019023), (43.24267516869766, 3594677), (43.24267516869766, 3142379), (43.24267516869766, 2463783), (42.100667253468956, 8478109), (41.64177063981604, 5989751), (41.372197942512756, 1816877), (41.365037944904195, 5193359), (40.86959688946432, 5005484), (40.86959688946432, 3951381)]}, {'query_id': '1114286', 'scores': [(26.74043873052966, 6846966), (26.74043873052966, 4956824), (26.571946729577576, 2589532), (26.46978385632458, 4956825), (26.02173778202728, 8720406), (26.02173778202728, 3960437), (26.02173778202728, 2177460), (25.944776379603326, 8413644), (25.944776379603326, 6040011), (25.86209312148852, 6269131)]}, {'query_id': '1114993', 'scores': [(31.87074657982275, 4104358), (30.981149493221995, 8113957), (30.467171891895834, 6612208), (30.467171891895834, 6203023), (30.467171891895834, 6011208), (30.467171891895834, 4956693), (30.467171891895834, 3271410), (30.467171891895834, 2086636), (30.467171891895834, 1881796), (30.467171891895834, 1724571)]}, {'query_id': '1115210', 'scores': [(56.23960159264233, 8111887), (51.838241468000774, 8111878), (45.85690591400068, 8111885), (45.85690591400068, 8111884), (45.85690591400068, 8111882), (32.99891697009785, 8390254), (32.99891697009785, 7440361), (32.23150029637465, 7440364), (32.23150029637465, 7440363), (31.498966198729775, 7440366)]}, {'query_id': '1116380', 'scores': [(51.55126145420928, 8100548), (51.427279378454784, 8100546), (51.427279378454784, 8100543), (40.805316694675206, 521341), (36.097010922212675, 4118339), (36.097010922212675, 1422757), (36.097010922212675, 137869), (30.22340281054382, 5104772), (30.22340281054382, 1645711), (30.22340281054382, 444969)]}, {'query_id': '1117817', 'scores': [(44.85787012481133, 5310910), (44.85787012481133, 3516296), (42.20441204717148, 4486487), (40.19114717744313, 7531176), (36.53236138663283, 6537918), (36.27688342434477, 1005145), (35.48705971968379, 1035058), (35.203186468620046, 109686), (34.67246691228346, 5970406), (34.67246691228346, 5447234)]}, {'query_id': '1117886', 'scores': [(45.155571001571154, 2921720), (43.84248561201732, 556178), (43.51556748799402, 8456607), (41.77729167614943, 7460949), (41.77729167614943, 7460948), (41.77729167614943, 7024723), (41.77729167614943, 5376945), (41.61933775281216, 5094343), (41.405913909284045, 5301093), (40.55085061517856, 8407176)]}, {'query_id': '1118370', 'scores': [(28.23447351263131, 8656398), (27.84183144059962, 7476349), (27.84183144059962, 2951456), (27.573872754238188, 2224024), (27.536156936053047, 8081645), (27.536156936053047, 2224022), (27.143514864021355, 6211291), (27.143514864021355, 3608410), (27.143514864021355, 1284328), (26.731574793473737, 861060)]}, {'query_id': '1118426', 'scores': [(42.41389325551132, 8081194), (42.41389325551132, 8081193), (36.39603637919815, 8081199), (36.39603637919815, 8081196), (36.39603637919815, 2424347), (34.77247446904838, 6622100), (34.77247446904838, 5598458), (34.77247446904838, 2424353), (32.80075099810084, 6208983), (32.26026676652181, 2469356)]}, {'query_id': '1119118', 'scores': [(47.24855375761641, 8074735), (46.37104402641371, 8074733), (39.56927285253072, 6026130), (39.56927285253072, 5472178), (39.56927285253072, 3609036), (37.732918441716485, 8074741), (37.732918441716485, 8074734), (31.927854066067788, 5451461), (31.927854066067788, 5185340), (31.927854066067788, 4246892)]}, {'query_id': '1119543', 'scores': [(73.14390257979977, 8070706), (67.26154423079493, 8070713), (63.71001486756422, 1603603), (63.49580937728193, 405831), (62.682088206903956, 405832), (61.90666894198526, 5052354), (61.678958097108485, 8070707), (58.42565670222966, 4770699), (57.7901822467058, 3612262), (50.40572856579277, 3612261)]}, {'query_id': '1120588', 'scores': [(50.74883407765331, 8060866), (50.034889642520575, 8060864), (49.07511407369297, 8060867), (47.98424289062399, 8060861), (46.949503514089606, 8060862), (46.65909800121558, 8060868), (44.18036897245296, 8060859), (43.48233885670176, 8060863), (43.48233885670176, 8060860), (42.447599480167376, 2313348)]}, {'query_id': '1121353', 'scores': [(57.75758307518821, 8053113), (56.43735242720638, 8053117), (48.66050593121671, 1854970), (45.295535427894414, 4872676), (44.20432528727399, 4210422), (44.20432528727399, 623601), (41.99125020017842, 6379159), (41.31466705546898, 5606064), (40.477130174718546, 5277905), (40.477130174718546, 937044)]}, {'query_id': '1121879', 'scores': [(63.179294306621756, 2603303), (63.16796157074404, 3024534), (61.838159871381336, 1307737), (61.506273146994516, 8048097), (61.506273146994516, 2603307), (61.506273146994516, 1406295), (57.133980265637504, 2193862), (57.02264321039648, 4995632), (56.92733381460421, 8048100), (55.609269659927776, 1025282)]}, {'query_id': '1122138', 'scores': [(27.919834320303984, 4342247), (27.8389152606343, 4342246), (27.79093190871536, 5871298), (26.811595024827888, 5463811), (26.519075572290216, 6283517), (26.25681570236712, 7067825), (26.25681570236712, 6963235), (25.766503771641332, 3967084), (25.766503771641332, 3647404), (25.766503771641332, 3345300)]}, {'query_id': '1122767', 'scores': [(55.379751805473646, 7544609), (55.379751805473646, 1575377), (54.0375198358854, 8039590), (54.0375198358854, 5877496), (52.61612157586279, 5877500), (50.630315605392155, 6851717), (50.630315605392155, 6179207), (50.630315605392155, 2859907), (50.0003116306419, 2019371), (48.08165371127525, 4550504)]}, {'query_id': '1122843', 'scores': [(77.38656103218096, 2332417), (77.360329323602, 2332424), (75.96491948664827, 8772747), (75.48098511711144, 225969), (73.60972450880568, 7998359), (72.72505845313061, 4947308), (72.43026526635228, 2332416), (71.94918042370892, 3880592), (71.85814009539307, 2332425), (71.58676217309764, 3312892)]}, {'query_id': '1123657', 'scores': [(63.052005507112, 8032346), (50.03639026539902, 8032342), (47.47299756079464, 8032349), (45.27116795141025, 7221025), (44.69577592931092, 5571459), (44.262960619391436, 8032348), (44.262960619391436, 8032347), (44.262960619391436, 8032343), (44.10126654142709, 7988749), (44.06405784924217, 5572500)]}, {'query_id': '1124552', 'scores': [(55.44700893803659, 5782491), (46.36459394400007, 3442729), (44.06691367266147, 4743535), (44.06691367266147, 3442725), (42.75821015325354, 7082651), (42.642721094215034, 8214271), (42.55223627018191, 2601066), (42.252142132792564, 2601070), (42.252142132792564, 1797406), (41.691096571586385, 6263843)]}, {'query_id': '1125632', 'scores': [(63.47869767216254, 7011199), (63.32635300623171, 6721396), (63.078436368782235, 1278069), (62.82611482802085, 7191566), (62.82611482802085, 1475286), (62.82611482802085, 1243662), (62.68867946320094, 6934400), (62.68867946320094, 6543904), (62.041313061301025, 7011204), (62.041313061301025, 5847127)]}, {'query_id': '1125755', 'scores': [(32.17472862571786, 1744931), (30.980582314378243, 7420642), (30.980582314378243, 2206347), (30.276862487099784, 7856574), (30.276862487099784, 3808440), (29.98851592373698, 7254544), (29.08271617576017, 7387332), (29.08271617576017, 7254547), (29.08271617576017, 2938068), (29.08271617576017, 482655)]}, {'query_id': '1126523', 'scores': [(43.40952262665484, 8805840), (42.215376315315225, 8805844), (42.215376315315225, 8805841), (38.40073155434851, 8805837), (33.06025459504485, 8805846), (33.06025459504485, 8556233), (33.06025459504485, 3214173), (33.06025459504485, 3025043), (33.06025459504485, 2841635), (29.245609834078138, 2774244)]}, {'query_id': '1126738', 'scores': [(73.44858039702677, 8790091), (73.44858039702677, 1091656), (71.97337726278052, 1091662), (71.97337726278052, 1091658), (66.09775988925377, 8790089), (66.09775988925377, 1091654), (64.69687693845499, 8790093), (64.69687693845499, 8790090), (64.69687693845499, 8790088), (62.64053317556626, 8790087)]}, {'query_id': '1127004', 'scores': [(50.772228502850425, 6481277), (50.50644958893912, 6531608), (50.34667475165446, 695631), (50.343155707666256, 5224906), (50.08089583774316, 1756214), (49.9770828639771, 3961675), (49.85636282092867, 6011954), (49.714822994054, 860288), (49.60403746263563, 6654716), (49.60403746263563, 4964550)]}, {'query_id': '1127233', 'scores': [(33.43839401182032, 8100081), (33.12034911178168, 8100077), (32.94702305514685, 8527129), (32.79347511389552, 5400207), (32.67990674569691, 6611628), (32.565921152783524, 124809), (32.47666828286078, 6438836), (32.47666828286078, 3842712), (32.47666828286078, 124808), (32.35852063029373, 7619778)]}, {'query_id': '1127540', 'scores': [(36.59702052801311, 8726963), (36.59702052801311, 8726962), (36.59702052801311, 186476), (35.01607652466533, 8726961), (35.01607652466533, 8726956), (35.01607652466533, 8726954), (32.732419360013246, 2574460), (32.732419360013246, 2044372), (32.732419360013246, 2044369), (32.732419360013246, 2044368)]}, {'query_id': '1128456', 'scores': [(67.20456401098838, 8649622), (57.76436548388226, 1459777), (50.76898220701672, 7526299), (49.37914260305433, 4084616), (48.303610569315055, 8649615), (48.303610569315055, 3721820), (45.85108822376522, 4071160), (45.85108822376522, 3121928), (45.85108822376522, 3121927), (45.85108822376522, 2534794)]}, {'query_id': '1129081', 'scores': [(37.17436875892819, 8599986), (32.885018517513394, 7871513), (32.885018517513394, 6962228), (32.885018517513394, 6794362), (32.885018517513394, 4178203), (32.885018517513394, 3762080), (32.885018517513394, 2770798), (32.885018517513394, 2156392), (32.885018517513394, 1044198), (32.885018517513394, 687772)]}, {'query_id': '1130705', 'scores': [(29.689061772055993, 1468216), (29.54307887055976, 8442374), (29.54307887055976, 1621163), (29.440090933202825, 6762105), (29.440090933202825, 6762102), (29.30440841255797, 6628960), (29.30440841255797, 6233460), (29.30440841255797, 5379660), (29.30440841255797, 3570934), (29.117524940726383, 1910013)]}, {'query_id': '1130734', 'scores': [(63.84021675814829, 2546803), (62.24989648942528, 7212169), (61.22288170736921, 8460898), (61.107585650625595, 8460901), (60.75301540282305, 8460902), (56.01056272100301, 4311761), (56.01056272100301, 2810717), (56.01056272100301, 1081476), (55.53790282604993, 3823605), (54.65571400274294, 7427232)]}, {'query_id': '1130847', 'scores': [(67.05464563572143, 5474954), (63.94480815670349, 4201753), (62.16186061328391, 7332266), (62.16186061328391, 7332262), (61.340966056268755, 7332261), (61.1328327729, 6209899), (60.864310037527815, 8299223), (60.23497606629693, 8299227), (60.118149081715956, 1458056), (58.931600446827275, 6149589)]}, {'query_id': '1131069', 'scores': [(55.15551722305412, 8432048), (54.24704755495573, 8432050), (52.14839993614961, 8191743), (52.14839993614961, 1780971), (52.14839993614961, 1780963), (45.691755539423795, 6907802), (45.13610475303173, 1780966), (44.2794966449483, 6809701), (44.2794966449483, 6809698), (43.90531916252551, 6809699)]}, {'query_id': '1132044', 'scores': [(52.26632639511176, 5858455), (51.334243825644954, 1014774), (50.643211669671665, 5858453), (50.57880414504043, 2121532), (48.08817157388742, 7645045), (47.79800040884696, 1691946), (46.306366919840315, 1818480), (46.14078024084789, 6166188), (45.98443939626003, 1650397), (45.79050502439325, 5810405)]}, {'query_id': '1132247', 'scores': [(72.1955852790043, 6246993), (68.80605331931017, 1210363), (68.80605331931017, 1210358), (68.80605331931017, 1210357), (66.74093641141356, 7390343), (65.3982141842798, 7390342), (65.3982141842798, 6525827), (65.39718289242735, 8238483), (64.6102215584632, 4623273), (63.953129242230176, 7390347)]}, {'query_id': '1132532', 'scores': [(59.60401572783728, 23966), (56.61600785589939, 6994907), (52.94313594278585, 4893092), (51.99571830273818, 6832793), (51.98341929772242, 8374347), (51.78606666869286, 3299150), (51.78606666869286, 2762578), (51.68063277226257, 5989422), (50.82947348367798, 1976995), (50.13695259787546, 6832795)]}, {'query_id': '1132842', 'scores': [(60.856756741885064, 3158781), (58.39076225103622, 7364932), (57.95182109155522, 834533), (57.3113316585592, 6536692), (57.22424214109628, 6570513), (56.85987297446198, 4874198), (56.75267792105353, 4132635), (56.533557076033645, 2324752), (56.30999901697635, 2941398), (53.63309923461685, 2391268)]}, {'query_id': '1132943', 'scores': [(48.424366336331346, 423711), (47.855090450069376, 5552077), (47.80704214079758, 6253953), (45.54683476242134, 622662), (45.247760326156964, 3437105), (40.689922629440574, 6469809), (40.262600755389386, 5782177), (40.13237429703533, 2107961), (40.124596856585875, 2248898), (40.124596856585875, 2107962)]}, {'query_id': '1132950', 'scores': [(40.64063818226049, 8274826), (39.48926366135125, 8619165), (38.954800117006194, 5867940), (37.83142796540978, 8274822), (37.83142796540978, 2303442), (37.39777270313935, 2132514), (37.39777270313935, 261777), (37.353298383953, 5845390), (36.58693749011307, 6590922), (36.2792718712176, 6625752)]}, {'query_id': '1133485', 'scores': [(38.68264716789245, 5221748), (38.53758164410002, 8330714), (38.51171756975086, 3156351), (38.474855652657965, 3183897), (38.45358261098119, 2418248), (38.42309613271226, 6195711), (38.38805974627046, 8692792), (38.34727342929999, 5630061), (38.33293869903945, 4090344), (38.144682510779624, 3089617)]}, {'query_id': '1133579', 'scores': [(46.19416112309672, 8220878), (45.67825283033753, 4931644), (44.44967034015408, 1817055), (43.48514614064777, 4931642), (43.48514614064777, 4931641), (41.0367385572856, 670206), (39.952171119483616, 8220879), (39.952171119483616, 5335041), (39.952171119483616, 3691486), (39.952171119483616, 2448461)]}, {'query_id': '1134094', 'scores': [(43.76191857295562, 8172161), (41.25123084186529, 8172166), (41.25123084186529, 7182375), (39.90516798705288, 6426847), (39.90516798705288, 6426846), (39.90516798705288, 3445457), (38.64233377456488, 8172159), (36.48823920175982, 7182378), (36.48823920175982, 6426844), (36.48823920175982, 5357491)]}, {'query_id': '1134207', 'scores': [(27.84220368943989, 1698802), (27.777080376791655, 374550), (27.371782415275803, 7155676), (26.98490708837431, 5592724), (26.792907708165075, 8817805), (26.792907708165075, 853685), (26.732039707896472, 3225711), (26.702302643158767, 7049025), (26.418520741440894, 8246744), (26.26161843373238, 3362951)]}, {'query_id': '1134431', 'scores': [(60.329909957113465, 3128065), (58.98846276501494, 8141222), (57.99616841203113, 8141223), (56.646980063516, 8141221), (56.578934153071366, 7974008), (54.571988313836954, 8141220), (50.512023948032635, 5010274), (50.386254816336695, 2339181), (50.24871688092816, 6720464), (50.24871688092816, 3243219)]}, {'query_id': '1134680', 'scores': [(81.37645784571794, 8118937), (70.29330432308248, 8118936), (67.93812033206164, 8118930), (57.90424061179279, 8796589), (57.46690544186423, 8307236), (57.072900122153335, 8063270), (57.072900122153335, 3526958), (57.072900122153335, 2456858), (57.072900122153335, 2200409), (57.072900122153335, 1865327)]}, {'query_id': '1134939', 'scores': [(42.853951301835025, 322652), (42.82268562839048, 8093333), (41.41079790679133, 8093330), (36.63262891754617, 6204714), (36.63262891754617, 2090588), (33.22861335484515, 8455233), (33.22861335484515, 8455232), (33.22861335484515, 7543137), (32.47341759678049, 8093327), (32.47341759678049, 7543128)]}, {'query_id': '1134988', 'scores': [(80.10000590743692, 8088407), (77.98896099062691, 8088409), (74.36393657919554, 380755), (73.71837614401221, 8088404), (70.0404375827346, 380753), (70.0404375827346, 380752), (69.86915343009824, 7296294), (69.6446537262785, 380748), (69.00493401986287, 6432242), (67.9293926659246, 380749)]}, {'query_id': '1135268', 'scores': [(43.49021735717509, 8386270), (43.21097059091062, 2966472), (42.93358408437036, 5717144), (42.906508451832195, 6259848), (42.906508451832195, 6087160), (42.42816725484123, 4260), (42.373556842606064, 5225950), (42.373556842606064, 1683963), (42.211323044619604, 8371436), (42.033828410392914, 3622462)]}, {'query_id': '1135283', 'scores': [(35.10163189483498, 8061944), (32.67701976166231, 2582837), (31.993737003931642, 4502908), (31.517136554906347, 3368379), (31.381455976189663, 7770955), (31.381455976189663, 7043720), (31.381455976189663, 6979076), (31.381455976189663, 5959072), (31.381455976189663, 5834915), (31.381455976189663, 3989820)]}, {'query_id': '1135413', 'scores': [(69.42058685825441, 8050662), (66.59143012957217, 8050667), (60.01578563077492, 8050665), (59.707485362123684, 2115732), (58.229292786921036, 6672865), (57.9222117134223, 7522683), (57.55417662868983, 6039815), (57.55417662868983, 5287000), (57.55417662868983, 2061136), (57.45661105415273, 5118096)]}, {'query_id': '1135626', 'scores': [(51.93404238517209, 2003615), (51.62224336752075, 4035834), (51.580530116613886, 7676917), (51.55423480995148, 6532689), (51.27003752001791, 2902886), (50.29887206225635, 4619335), (49.39151033932774, 4892864), (48.74735233333061, 7714006), (48.64615955999031, 4079092), (48.43538885440741, 4035841)]}, {'query_id': '1136043', 'scores': [(48.49227419794402, 8728069), (48.426503167881336, 8728071), (46.5246059242415, 8728070), (46.49675513529931, 8728072), (45.50936051747823, 1056540), (44.248514810669484, 8728076), (44.145425304066265, 1056534), (43.471983580163986, 7018326), (43.471983580163986, 5326197), (42.89524230080494, 8728077)]}, {'query_id': '1136047', 'scores': [(58.98542692069104, 8724031), (56.61585013690356, 8724032), (55.22938011688501, 3988779), (54.35552697979545, 3597758), (54.35552697979545, 3006066), (53.232541218182696, 7009393), (52.08997302746144, 8328670), (51.82862348204492, 512482), (51.49548411356448, 2815435), (50.82993629738824, 8724036)]}, {'query_id': '1136769', 'scores': [(75.39595534457717, 8082043), (74.15151279910532, 189017), (72.32974381400743, 129247), (71.31048175331118, 189019), (70.16068402993051, 8082050), (70.152494218761, 4713242), (69.19259156812069, 189016), (67.21298740028747, 8082046), (67.21298740028747, 5718075), (63.81640879032942, 5508127)]}, {'query_id': '1136962', 'scores': [(112.88596394931662, 7631079), (112.75965569938819, 6185710), (111.92902232390603, 6185711), (111.30390009177441, 6185717), (110.63352155344846, 8336574), (109.71769576945059, 6185714), (109.71769576945059, 6185712), (103.55176921943426, 596001), (100.19062985407479, 6185716), (99.81769162303566, 4601613)]}, {'query_id': '118440', 'scores': [(38.08312964079179, 7035722), (38.08312964079179, 1569786), (36.56341517398157, 8319567), (35.918473572157644, 4737856), (35.918473572157644, 1569787), (34.7354464060783, 8412702), (32.1592702708219, 6750552), (32.1592702708219, 6750548), (31.476644250922377, 6750547), (31.476644250922377, 2975214)]}, {'query_id': '119821', 'scores': [(42.15927546938831, 7093228), (39.9736450641465, 7172227), (39.9736450641465, 6527223), (38.552187189210215, 8012784), (38.552187189210215, 6267963), (38.552187189210215, 3365586), (38.552187189210215, 259724), (33.44246188444183, 8012780), (32.682405932522705, 2212944), (31.261431761543456, 6072161)]}, {'query_id': '121171', 'scores': [(39.900354287263255, 8260192), (34.54604464183217, 7165126), (34.54604464183217, 6930941), (34.54604464183217, 2337570), (29.631480688369646, 451171), (29.479264177984188, 8643275), (29.479264177984188, 451173), (29.002510579418402, 2550811), (29.002510579418402, 451172), (29.002510579418402, 259437)]}, {'query_id': '125659', 'scores': [(43.57127492663355, 6976559), (43.57127492663355, 4522348), (43.57127492663355, 1335471), (42.12750378945115, 829647), (41.81379309610916, 5881859), (41.245034769340336, 8290885), (41.245034769340336, 6635625), (41.245034769340336, 2745708), (41.245034769340336, 1106134), (39.80126363215793, 5693282)]}, {'query_id': '135802', 'scores': [(44.30502903866332, 8692623), (43.110882727323705, 8692626), (42.82869160357404, 8692628), (41.63454529223443, 8692619), (41.63454529223443, 6037338), (41.63454529223443, 6037331), (37.88691949546934, 8731214), (37.88691949546934, 7327478), (37.88691949546934, 5394718), (35.816350651275435, 8692624)]}, {'query_id': '141630', 'scores': [(74.23435832367451, 2473236), (70.95430172369446, 8319991), (70.79139859818275, 6290795), (69.17287093978382, 8319989), (68.7532413262578, 7985119), (68.45988473421217, 2366121), (68.23941274199179, 8319987), (68.07681391968251, 3500851), (67.75577831700812, 3500859), (67.38377377919639, 6496223)]}, {'query_id': '144862', 'scores': [(47.03296268584424, 92239), (46.85451620671749, 92237), (46.41977710013721, 92238), (46.303802544594035, 2865950), (45.841647474804866, 92233), (45.04799952283608, 5292762), (44.91752322502883, 6150954), (44.57574445755628, 8320087), (44.57574445755628, 7245936), (44.57574445755628, 92241)]}, {'query_id': '156498', 'scores': [(61.9888765320714, 4273410), (53.085465718926976, 5606985), (52.187491504131046, 8823842), (51.484728624873746, 3288004), (51.18236403261821, 5043014), (50.99490726964783, 7075146), (50.99490726964783, 3397582), (50.7483870606257, 3397578), (50.68386759140607, 3301559), (50.47597954773579, 3288005)]}, {'query_id': '166046', 'scores': [(56.70604024220032, 7947463), (56.70604024220032, 1076792), (56.70604024220032, 1076791), (55.3531630803186, 1076793), (51.515912760751235, 1076795), (51.515912760751235, 1076794), (46.37415447435012, 96920), (45.87658876205094, 7947466), (44.98110483922546, 96927), (44.64407493868207, 7865285)]}, {'query_id': '169208', 'scores': [(46.1293108649256, 4859433), (45.25272675187287, 4859432), (45.17568359612078, 4580137), (45.044664136325935, 4762140), (44.6702035085016, 7251639), (44.451314500235696, 4202275), (44.33368093653263, 1395417), (44.15687115058914, 85987), (43.920419240465975, 2888366), (43.91192172575532, 4762142)]}, {'query_id': '174463', 'scores': [(44.757460801514235, 8304456), (44.14240235360076, 4501665), (42.6438588329258, 102528), (39.75116285018439, 6294949), (39.567975704697155, 7089630), (39.567975704697155, 7089629), (38.740159850529935, 8304458), (38.420287007787394, 7402420), (36.520498788441714, 7611683), (36.520498788441714, 5793989)]}, {'query_id': '175920', 'scores': [(91.71097317824558, 8234235), (80.44016322758429, 101345), (71.7350906626466, 8537849), (71.55857417490498, 7615731), (71.55857417490498, 5673665), (70.68012422444055, 8537850), (69.96805339615102, 7360509), (69.96805339615102, 6815325), (69.96805339615102, 5717848), (69.96805339615102, 5673664)]}, {'query_id': '177604', 'scores': [(49.127188538359235, 8451995), (47.6872321025401, 8451992), (43.262039574839875, 5627757), (43.23395161262104, 2226834), (43.01994243804827, 5034647), (41.636537468464745, 3444466), (41.636537468464745, 1308162), (39.29998151875854, 8451989), (39.08671906601341, 8451987), (38.74797180084217, 8451991)]}, {'query_id': '181626', 'scores': [(40.53255333688902, 43641), (39.81660366233994, 436351), (39.423961590308245, 7795335), (39.423961590308245, 4798089), (39.423961590308245, 2183331), (39.14320446662304, 71074), (38.68810690588239, 8069297), (38.478089053400325, 71073), (38.42725479207395, 5427354), (38.42725479207395, 4091013)]}, {'query_id': '197312', 'scores': [(41.162546692577244, 2317108), (41.162546692577244, 2317106), (40.70629139945948, 5823013), (40.01806817887634, 5823008), (39.95085483146295, 1179432), (39.58592337529094, 5823015), (39.58592337529094, 1179433), (39.53481117066497, 1665689), (39.32952437503172, 2317100), (39.32952437503172, 1239571)]}, {'query_id': '206106', 'scores': [(59.081627578287424, 6479522), (57.650528259341556, 75400), (57.06990662658893, 7079997), (56.49522468802833, 1390597), (56.232731249131675, 5093731), (55.87373213949044, 8427974), (55.6264222680681, 8737574), (55.43360929792247, 3217609), (55.43360929792247, 2947646), (55.00121408359372, 7118029)]}, {'query_id': '227873', 'scores': [(58.34724123600914, 7276796), (57.03242917058785, 8453473), (55.9009242815982, 7276798), (55.9009242815982, 3250369), (55.9009242815982, 2054488), (52.62294352267836, 7019128), (51.96420519159956, 3324178), (47.627275454116344, 1900788), (45.111732400332265, 8453475), (44.89234565470312, 8453476)]}, {'query_id': '23849', 'scores': [(41.68579344085422, 4834498), (40.61391470104821, 2674124), (40.4297704726566, 6728787), (40.34534166780886, 4348282), (40.229052936603495, 5888570), (39.78504141745572, 2373852), (39.31529238166785, 641196), (39.27346292800285, 8246990), (39.27346292800285, 2017213), (39.02916558598355, 5363400)]}, {'query_id': '240158', 'scores': [(58.09637454065858, 2043788), (56.44957479929517, 991184), (56.413675929163745, 6536692), (55.79646497250479, 1567595), (55.35762668220193, 4874198), (51.10190407357834, 2819781), (51.01242314410505, 2819779), (50.22028401002015, 3542898), (49.67444386045439, 2819784), (49.615693447155564, 6239283)]}, {'query_id': '245052', 'scores': [(54.78227259196777, 3828386), (48.75521741769647, 5023796), (47.995572749581626, 5878365), (47.975278823529166, 4205737), (47.975278823529166, 66616), (47.56453582649563, 8282910), (47.524744090736924, 5666135), (47.34034464993742, 5143412), (47.197713435129714, 8279354), (47.197713435129714, 3379257)]}, {'query_id': '246883', 'scores': [(58.49862076925101, 3050794), (57.60830943726493, 8699192), (55.91784080624278, 5050393), (53.65520307780655, 8699195), (52.6072760439256, 4959448), (51.86877037029821, 5683913), (51.86877037029821, 2413876), (51.444031264185185, 6242496), (51.11560305782305, 6083923), (51.02702559511082, 8420183)]}, {'query_id': '253749', 'scores': [(45.057801432053665, 6190136), (44.08063422763945, 3516374), (43.65190932842373, 4070966), (40.541906325052345, 3516370), (39.958790713231274, 3516368), (38.512951418593566, 8254833), (37.06755826560743, 8254894), (35.74893113763399, 8521273), (35.70780947966956, 6260689), (35.11229189839443, 6447304)]}, {'query_id': '256942', 'scores': [(61.19538851256883, 2602557), (51.801731127628024, 2364119), (51.03897487994514, 7213042), (50.680085431483825, 4890686), (50.28471879180756, 8327014), (50.22386515983437, 3262199), (49.96352603379433, 5297913), (49.69133710998485, 5083782), (49.69133710998485, 4604331), (49.69133710998485, 1183278)]}, {'query_id': '257119', 'scores': [(55.75933340374534, 8499023), (54.39758593496521, 4363515), (50.102035098427194, 4363509), (47.28987635244748, 8032136), (46.26756220282472, 3818024), (46.14063039509142, 889398), (45.75958434457088, 8276379), (45.58051708303541, 5853762), (45.572972626528454, 2112755), (45.54031879200133, 4212490)]}, {'query_id': '258062', 'scores': [(66.69137483880327, 8694161), (66.27035011023386, 4159145), (65.09502443451943, 18103), (63.328158808857125, 3683402), (61.77618395543402, 7918982), (61.21532207399008, 2707010), (61.103314797780655, 4174313), (60.30109614516119, 3195767), (59.61420479423781, 91493), (59.26003812686205, 4174307)]}, {'query_id': '26703', 'scores': [(37.324271411427155, 3532417), (35.531673443307156, 3532415), (35.531673443307156, 1485077), (30.284021724135023, 380356), (29.30558654255803, 7265225), (29.30558654255803, 1236259), (29.22874653709181, 3733548), (28.68822632325314, 3921274), (28.599411585419848, 3132961), (28.599411585419848, 524170)]}, {'query_id': '273695', 'scores': [(57.39333772315531, 8282608), (56.294011226722446, 5542707), (55.4143555432112, 7381998), (55.303084413960335, 5542705), (54.76567460544736, 2714742), (54.26834503742595, 7381994), (53.681698564261424, 5719948), (53.681698564261424, 4171983), (53.681698564261424, 413461), (53.6196640996621, 413460)]}, {'query_id': '302846', 'scores': [(67.08395635280485, 8418433), (64.9607030889068, 8418435), (61.47596110236128, 447099), (61.04943906015667, 8283558), (60.91228638190829, 6263014), (60.76169925588102, 3892531), (60.71801612755096, 3566357), (60.531106154491084, 5696466), (60.23256546751837, 447101), (59.93179017426158, 4234633)]}, {'query_id': '318362', 'scores': [(53.40244884319692, 5654804), (52.25139353733929, 3145979), (52.1992268837149, 429301), (50.640570936361115, 4299889), (49.86932284569948, 1714531), (48.99399023990718, 1714527), (46.9609011023055, 4299887), (46.83577965533793, 543937), (46.10061642160003, 519111), (45.40099425002858, 3145987)]}, {'query_id': '324585', 'scores': [(56.6735402484278, 1051441), (51.95359889081024, 8290314), (49.76506995256631, 8290310), (48.9037615977678, 3608325), (47.650925464522864, 7977807), (47.158096835675565, 7941125), (45.060602352130545, 7359510), (44.83921786558659, 8310496), (44.67331676706778, 405282), (44.44231889273502, 2194313)]}, {'query_id': '330501', 'scores': [(52.957923034839915, 6006326), (52.02650095566991, 3718639), (52.02650095566991, 1750071), (48.39151989218534, 6647777), (48.17758414622325, 7354312), (48.17758414622325, 4397700), (48.17758414622325, 452513), (47.887746603448946, 1899109), (47.887746603448946, 294036), (46.90178186118888, 2169021)]}, {'query_id': '330975', 'scores': [(75.05599580889046, 7908218), (68.74301967671714, 5750875), (68.61114697605723, 233948), (68.5330334751309, 4108178), (68.15973663074288, 8327008), (68.1111254177718, 3297858), (68.08899292900375, 233945), (67.74974611326772, 3297857), (67.3996258895076, 1097861), (67.33747323942067, 8541461)]}, {'query_id': '332593', 'scores': [(71.8431727999029, 8434873), (70.96516460501167, 5241323), (69.49640767907593, 5241322), (68.0112050334011, 7073995), (66.79701737709765, 4431502), (66.79701737709765, 1650687), (66.16236077687137, 8434869), (66.16236077687137, 7073996), (65.09407417923123, 8434872), (64.27459050314303, 4614346)]}, {'query_id': '336901', 'scores': [(60.34176644669731, 3646550), (59.07907523247901, 2909600), (58.29768973567842, 8528674), (57.95948010645201, 8130980), (57.288980100773244, 7836261), (56.836838589485986, 8528681), (56.836838589485986, 8528680), (56.836838589485986, 8528679), (56.836838589485986, 8528678), (56.836838589485986, 8528677)]}, {'query_id': '3505', 'scores': [(40.22358491932966, 8285655), (40.11525636208674, 8285658), (39.88033014530177, 4520880), (39.84714658497347, 5469617), (39.70094060049665, 8285654), (39.70094060049665, 8285653), (39.6872449489565, 8285656), (39.60667360349221, 502056), (39.599433895422756, 1030851), (39.500524053118625, 1076196)]}, {'query_id': '360721', 'scores': [(54.08096923037398, 6318492), (49.124358954616284, 4284999), (49.00183996322588, 101662), (48.53865442415907, 7043084), (47.770496745609805, 101666), (47.473747434104666, 7826496), (47.17872623267921, 7826493), (47.09467482435841, 1750204), (46.36881422953773, 7826491), (46.24284593278091, 6206175)]}, {'query_id': '384356', 'scores': [(59.99480648980964, 7565188), (46.92340940020695, 2032126), (46.481273719188664, 2032127), (46.020136129606584, 3705729), (45.91836083601846, 5381456), (45.87978014546364, 6863040), (45.861035716616236, 4648730), (45.861035716616236, 412129), (45.83216732113237, 6543899), (45.83216732113237, 2032125)]}, {'query_id': '390360', 'scores': [(53.418343993623616, 604111), (49.684925429118564, 5423359), (49.00225594332635, 2203877), (49.00225594332635, 2203875), (48.35814270952878, 7679050), (47.49045009446167, 6179316), (46.37950467622069, 4683960), (46.37950467622069, 324991), (45.98686260418899, 324994), (45.98686260418899, 324992)]}, {'query_id': '405163', 'scores': [(39.65672194854834, 851536), (38.488027157913606, 6289824), (37.53346868465028, 7622492), (37.2041996029985, 429247), (37.2041996029985, 429246), (37.2041996029985, 429245), (35.08094633910045, 3293026), (35.08094633910045, 2748170), (35.08094633910045, 1518541), (35.08094633910045, 1352548)]}, {'query_id': '42255', 'scores': [(81.99830230475472, 8746197), (76.93941336033014, 8746189), (72.53274575453885, 8746188), (70.56698109788343, 8746194), (67.6566054266488, 3723425), (67.61258658137763, 3063186), (67.26710938112652, 2882385), (67.18728194459419, 8746196), (67.13692256262429, 7123389), (67.06107543765981, 6784918)]}, {'query_id': '425632', 'scores': [(54.9849127050137, 8454642), (54.9849127050137, 8454639), (52.96210694052134, 8454636), (50.669009221373585, 8454635), (50.669009221373585, 650345), (39.10470126086036, 8454641), (33.088593374574145, 8454643), (33.088593374574145, 8454640), (33.088593374574145, 8292459), (33.088593374574145, 6843069)]}, {'query_id': '426175', 'scores': [(36.62081382717152, 330964), (35.82853475260643, 6209130), (35.82853475260643, 4362255), (35.60355811971062, 6758532), (35.60355811971062, 5030738), (34.74455643047207, 6049564), (34.74455643047207, 3017346), (33.89151676948499, 6016624), (33.75483709287114, 3870458), (33.75483709287114, 3562170)]}, {'query_id': '42752', 'scores': [(39.14564253381385, 4303156), (39.03660305752797, 1254236), (39.02441568362146, 7920885), (39.01113857105172, 8660833), (39.01113857105172, 5932129), (38.709879782131125, 5325520), (38.61417208188671, 3818076), (38.60023144801321, 4821761), (38.36203376852673, 4747962), (38.34406982641851, 1986201)]}, {'query_id': '435548', 'scores': [(34.23124563780922, 8547993), (34.23124563780922, 8547992), (34.176488231257274, 7875423), (33.63365758977925, 7875421), (33.110041576373746, 7080768), (33.110041576373746, 235984), (33.063232634841555, 7875428), (33.063232634841555, 6064445), (33.063232634841555, 1394884), (31.996785979958027, 6424374)]}, {'query_id': '436707', 'scores': [(39.40572489481224, 1595287), (38.74514272491306, 2203550), (38.74514272491306, 2101427), (38.74514272491306, 1825031), (38.21323689906292, 5413961), (38.21323689906292, 3370495), (37.41170992684386, 5044105), (36.999759545909825, 8641216), (36.999759545909825, 8200621), (36.79010061606555, 6045618)]}, {'query_id': '444389', 'scores': [(42.94186775902429, 5896071), (42.046252432995615, 8495210), (35.83137367113159, 8044142), (35.83137367113159, 7705876), (35.83137367113159, 5208202), (35.32449516813813, 1722558), (35.05032284274404, 7484162), (35.05032284274404, 7150173), (35.05032284274404, 1048029), (34.60998547165657, 5251228)]}, {'query_id': '449367', 'scores': [(44.8341386768903, 8432230), (44.72002606516108, 635517), (44.14524858103866, 6632652), (43.897347438047056, 6632645), (43.897347438047056, 6632644), (43.141947476924145, 8600273), (42.23984900445052, 5717147), (41.97108294868582, 2596719), (41.908055164774275, 386955), (41.658852702133046, 1998108)]}, {'query_id': '452915', 'scores': [(48.36946137833745, 135840), (46.890192374031805, 2447912), (46.87738946087531, 3332642), (46.531936960579785, 1044828), (46.24523279493054, 953667), (46.041625029853996, 1364570), (45.654370036914926, 8366603), (45.654370036914926, 8366600), (45.654370036914926, 3779965), (45.65074888077799, 7261641)]}, {'query_id': '463271', 'scores': [(53.29601843507277, 6239584), (52.39685443029627, 8604216), (52.39685443029627, 8326637), (52.39685443029627, 6239587), (52.39685443029627, 3914186), (51.53270374689229, 2455683), (51.217919239076224, 8326635), (51.05730677291079, 8435508), (51.05730677291079, 3914192), (50.719716451534886, 6005616)]}, {'query_id': '469589', 'scores': [(62.912578743328915, 8581474), (59.39705823591213, 8581470), (59.39705823591213, 8581468), (55.91797647853005, 2210168), (53.1220227840372, 4871853), (52.514692852944044, 2210175), (49.756696945673234, 8581475), (49.756696945673234, 8581466), (49.756696945673234, 2210172), (49.09149857461225, 6441572)]}, {'query_id': '47210', 'scores': [(72.04149698761682, 2042918), (71.17108161474187, 6180504), (70.45623705082295, 2338385), (68.23727637044001, 5207329), (68.20640307292012, 2545998), (66.95700332559241, 4748997), (65.852782818801, 3858013), (65.55637888637293, 2546002), (65.55637888637293, 2546001), (65.51196960857791, 7367873)]}, {'query_id': '482726', 'scores': [(25.30923542899766, 8730812), (24.957724524421835, 8572751), (24.43052399235153, 6318274), (24.43052399235153, 4734241), (24.43052399235153, 22062), (24.409738663233938, 1272524), (24.262889992818895, 4734239), (24.13162940827919, 4768911), (24.13162940827919, 3512937), (24.13162940827919, 1985718)]}, {'query_id': '48792', 'scores': [(60.142461202325876, 64048), (59.550091466651345, 64047), (54.50735422962718, 64041), (36.69575604269064, 6619302), (36.036718282285875, 5803986), (36.036718282285875, 3820076), (35.72228819898634, 3820082), (35.640934425829776, 6152218), (35.640934425829776, 3820080), (35.6362873153886, 6152219)]}, {'query_id': '50122', 'scores': [(52.51562336202875, 8273511), (49.597684585000025, 1417366), (48.859971433287924, 8273514), (48.413909775096556, 8273517), (48.413909775096556, 425197), (45.408942949956966, 7341000), (39.2229387623488, 8361668), (38.954794166429075, 8337928), (38.954794166429075, 5999140), (38.30410906282784, 8273519)]}, {'query_id': '514096', 'scores': [(26.050940719543505, 4786295), (25.780814184077244, 6286608), (25.780599403454357, 6474925), (25.780599403454357, 6285508), (25.634757394551883, 1837337), (25.56713868266199, 3534826), (25.42108189313663, 8460993), (25.42108189313663, 5936453), (25.400181338647513, 7016672), (25.400181338647513, 3708394)]}, {'query_id': '519025', 'scores': [(33.48802314850731, 4232478), (33.36552953940948, 288148), (32.752431387102305, 2692126), (32.589137505829434, 6805536), (32.589137505829434, 6526749), (32.32687763590634, 6526750), (32.32687763590634, 3394216), (32.32687763590634, 1697202), (32.2954207165846, 6236703), (32.2954207165846, 5931056)]}, {'query_id': '53233', 'scores': [(58.13944861859299, 8333467), (56.41224231051203, 8257054), (56.040393880016076, 8333470), (56.040393880016076, 8333463), (55.34519898821928, 8257048), (55.34519898821928, 5256809), (53.58047349365819, 8257051), (53.58047349365819, 7355044), (53.58047349365819, 5516316), (53.58047349365819, 579087)]}, {'query_id': '537060', 'scores': [(46.57353075342735, 8420027), (46.50644323832398, 8420024), (45.841199619032814, 8420032), (45.841199619032814, 8420025), (43.98220520095438, 8420030), (43.98220520095438, 8420029), (43.98220520095438, 8420026), (43.98220520095438, 7386519), (43.98220520095438, 1086305), (42.41082485029976, 8420031)]}, {'query_id': '537817', 'scores': [(51.42740150537853, 1956211), (51.28097565136481, 6665873), (51.182346891309884, 8527917), (50.997735771926855, 540370), (50.97572565962132, 5264609), (50.832582681357465, 1956207), (50.154353192274726, 3759425), (50.11052674147325, 540368), (50.06688091616198, 2025709), (49.87111331283676, 1095538)]}, {'query_id': '543273', 'scores': [(56.49100254755588, 8507214), (55.259922074494156, 8507211), (52.57248800654146, 8507209), (52.12979659723385, 4308075), (50.90665886078871, 5990522), (49.84291110147114, 8507212), (49.26606709208937, 8507208), (49.26606709208937, 4308073), (49.26606709208937, 3143117), (47.80177076058182, 4067764)]}, {'query_id': '545355', 'scores': [(65.28315715120661, 4290924), (63.222605024751985, 4290920), (61.91094842520934, 8547218), (61.91094842520934, 8547215), (61.60240395507491, 8547219), (61.60240395507491, 4290921), (61.025559945693146, 4290922), (55.4484455142361, 4290923), (48.33499173929434, 4290918), (48.33499173929434, 3294315)]}, {'query_id': '555530', 'scores': [(49.98257931748847, 1897716), (49.105465273627246, 7475084), (48.983391602053935, 4296807), (48.3441786563642, 4137963), (48.20351149968182, 1754575), (48.159110956669025, 6485976), (47.58397897626271, 6150835), (47.56191474971767, 3948935), (47.16302005438847, 4238674), (46.871775018971874, 1483513)]}, {'query_id': '583468', 'scores': [(36.90751119221192, 7491704), (36.76461198511346, 8326078), (36.543671772746784, 7491700), (36.543671772746784, 1675562), (36.41882313250726, 7706119), (36.41882313250726, 2267054), (36.41882313250726, 2267052), (36.41882313250726, 1675564), (36.31500209355011, 6654310), (36.31500209355011, 2080306)]}, {'query_id': '586148', 'scores': [(42.40527172830462, 5056079), (41.09699439133204, 960743), (40.97360954601184, 8271038), (40.89417475961996, 6148926), (40.14103736768357, 7779366), (39.96540013342907, 6854043), (39.96540013342907, 1658943), (39.9382177359715, 2811037), (39.86967718469157, 8567688), (39.805547007565096, 5678812)]}, {'query_id': '590019', 'scores': [(40.20411617629251, 4408763), (40.081109370110084, 2831848), (40.07505796133358, 2831847), (40.00770821803319, 632727), (39.50111262738527, 5075460), (39.23172777241244, 3453506), (39.17817875019908, 3266922), (39.12452349685708, 6066327), (39.11583359074084, 1248322), (39.10995876299576, 8369826)]}, {'query_id': '605127', 'scores': [(54.11548807064422, 1137979), (53.63868742978599, 8642036), (51.67908714869748, 8594907), (51.67908714869748, 1137978), (51.10513675116847, 5121398), (49.3849181350089, 3679424), (48.58270565493352, 8547782), (48.110446601436536, 8642038), (48.110446601436536, 1137982), (46.661053204929736, 6888324)]}, {'query_id': '610265', 'scores': [(57.81068897830704, 474853), (57.23985767433712, 222769), (57.22804056936947, 2877322), (57.22804056936947, 683818), (57.03437888331837, 2877320), (57.01526786569832, 1148168), (56.9202528010967, 4758657), (56.90115476305082, 1148169), (56.85419722015643, 7009719), (56.7922864372504, 2039693)]}, {'query_id': '611953', 'scores': [(76.86529178866935, 484756), (67.09434336461452, 7197690), (61.0807182461033, 3562872), (61.0807182461033, 484758), (60.15195191705084, 456354), (58.04387977170319, 8062950), (58.01152405574676, 456353), (57.50945951221826, 456359), (57.34150813202862, 8062949), (56.58132151609455, 8371935)]}, {'query_id': '640502', 'scores': [(44.100844821861514, 3551380), (43.81058170243895, 5827712), (43.348728971708255, 5827713), (42.467291970351525, 6730887), (42.467291970351525, 3271456), (42.467291970351525, 1861075), (42.44843004963134, 6985038), (42.42397995024684, 5150987), (42.2563746756901, 3678130), (42.14079192459329, 77134)]}, {'query_id': '64647', 'scores': [(49.53074057816176, 8293649), (48.95272005223479, 8293652), (45.97457718718075, 4161653), (43.778861021610396, 8293648), (36.55368203514877, 8599633), (35.33103896278777, 1422031), (35.26874016407929, 6508277), (34.940312891418245, 7509908), (34.940312891418245, 5692182), (34.940312891418245, 1310940)]}, {'query_id': '653399', 'scores': [(42.082893004833, 1969533), (40.02014293866196, 7133996), (39.333402948297895, 8458007), (39.17807835386691, 7485466), (39.17807835386691, 6343750), (39.17807835386691, 3111946), (39.17807835386691, 2007733), (38.5979610792473, 1674561), (38.36022516634597, 2399252), (38.287226667215904, 7092040)]}, {'query_id': '655526', 'scores': [(44.74561000299681, 6040076), (43.480011674325894, 8425101), (42.082366662207335, 206999), (38.463087250365206, 6707899), (38.463087250365206, 3864910), (38.463087250365206, 3021665), (38.463087250365206, 2105608), (34.030331576754556, 1826238), (33.55548974079984, 5883282), (32.79286497396349, 7135528)]}, {'query_id': '655914', 'scores': [(36.017649070385524, 8613490), (34.92954531739092, 1270195), (32.16743579055848, 8613495), (32.16743579055848, 8058173), (32.16743579055848, 5866957), (32.16743579055848, 5235329), (32.16743579055848, 2789189), (26.533999952542953, 3209412), (26.397695158266195, 6248270), (26.397695158266195, 2519335)]}, {'query_id': '660198', 'scores': [(34.829731126439555, 2845619), (33.25390365950289, 523901), (33.23231698228326, 2410610), (32.93869516094842, 4434042), (32.86208509584191, 7327517), (32.86208509584191, 4264632), (32.86208509584191, 3556727), (32.41064196814557, 7327515), (32.41064196814557, 4264630), (32.41064196814557, 3604202)]}, {'query_id': '67316', 'scores': [(69.31739877796132, 6680964), (61.74008977957729, 7630885), (60.67639069017124, 6764671), (59.43372139062002, 6179107), (59.222137775613646, 4163705), (58.84570810567906, 7112302), (58.67993075090326, 4252546), (58.67993075090326, 3810719), (57.706332307261796, 8431637), (57.49030490561766, 1974375)]}, {'query_id': '673670', 'scores': [(31.677507808414084, 3607501), (31.37866339512716, 7257303), (31.37866339512716, 2240359), (30.940821580311432, 8008345), (30.940821580311432, 3607497), (30.940821580311432, 1905068), (30.940821580311432, 1398756), (30.237621089849814, 2240358), (30.237621089849814, 2240357), (30.237621089849814, 2240355)]}, {'query_id': '701453', 'scores': [(44.37313410233498, 4355647), (44.37313410233498, 4330679), (43.7125107955118, 4355649), (43.5450485903323, 8626149), (43.20217902648689, 6999087), (43.001632677822585, 4330677), (42.59095006563356, 2036081), (42.50419993502207, 4355645), (42.39368264533894, 2036076), (42.248080501788145, 4330673)]}, {'query_id': '703782', 'scores': [(40.078870189466954, 4061595), (39.38949832640834, 4061598), (38.59522204940601, 4061597), (38.59522204940601, 344218), (38.5905197888808, 185599), (38.05294653000127, 4647484), (37.95344897359844, 4948640), (37.82236489135344, 3490915), (37.82236489135344, 611402), (37.641815117695224, 7687244)]}, {'query_id': '708979', 'scores': [(37.34653392819264, 6157147), (37.27320518594547, 4805197), (37.18849165746205, 279265), (37.11570850987285, 3811546), (37.08056894309277, 2889574), (36.96273068173775, 6313188), (36.93838504955863, 1447346), (36.89632375002151, 8088373), (36.85480796736847, 6409763), (36.85480796736847, 3422810)]}, {'query_id': '716113', 'scores': [(41.48546173539723, 8520612), (41.48546173539723, 8520611), (40.906594827461454, 8520617), (38.238773425670495, 8520618), (38.238773425670495, 8520614), (38.238773425670495, 8520613), (33.82660726117005, 8520616), (33.82660726117005, 8520610)]}, {'query_id': '730539', 'scores': [(51.34875866378623, 8577877), (49.11445004172529, 8276218), (48.31982754625078, 2666444), (47.85975559080795, 2666436), (46.41412937562057, 8577880), (42.74446282937568, 8577876), (42.74446282937568, 5428552), (42.74446282937568, 4605022), (33.249402949309726, 8276217), (33.249402949309726, 7916064)]}, {'query_id': '735482', 'scores': [(38.41575399332912, 8405801), (37.94984954029286, 8405802), (37.42754360349872, 4823368), (37.10116164799611, 8405807), (37.10116164799611, 338960), (36.70591114879592, 8405806), (35.87446505968181, 8405805), (33.84569762516202, 4117288), (33.763871141161395, 8405804), (33.763871141161395, 8405803)]}, {'query_id': '735922', 'scores': [(38.287478463923726, 5537112), (38.287478463923726, 5537109), (36.87285885862202, 5854095), (36.87285885862202, 281701), (36.87285885862202, 281697), (36.87285885862202, 180247), (35.84158648258619, 8626887), (35.28431209261885, 7371194), (35.28431209261885, 5861444), (35.28431209261885, 1432224)]}, {'query_id': '75198', 'scores': [(44.5680297652721, 7797928), (44.19144063474392, 2579716), (43.46729348042474, 8256960), (41.746862365439476, 6862216), (40.79090534179102, 6862217), (40.79090534179102, 3916717), (40.79090534179102, 2140526), (40.79090534179102, 2140522), (38.58286790175765, 7088445), (38.58286790175765, 2932859)]}, {'query_id': '768208', 'scores': [(37.75961602044883, 52162), (37.2327376573728, 52155), (36.3865390742507, 8525681), (36.3865390742507, 52159), (34.8045156362398, 3515884), (34.8045156362398, 3515882), (34.8045156362398, 813102), (34.8045156362398, 52160), (34.8045156362398, 52158), (34.8045156362398, 52157)]}, {'query_id': '779302', 'scores': [(47.82895060153685, 1583173), (46.79323829560066, 8768560), (46.79323829560066, 8768554), (44.164122442070315, 8768563), (44.164122442070315, 8768562), (44.164122442070315, 8768561), (44.164122442070315, 8768558), (44.164122442070315, 8768555), (36.72228991713043, 2411546), (36.161380922321825, 4638389)]}, {'query_id': '792635', 'scores': [(38.43586318271924, 4294317), (36.76473869651405, 8682263), (36.76473869651405, 8682259), (36.76473869651405, 8682258), (36.76473869651405, 8682254), (36.76473869651405, 6306494), (36.76473869651405, 6306491), (36.76473869651405, 4294318), (36.76473869651405, 4294312), (36.76473869651405, 4294311)]}, {'query_id': '794223', 'scores': [(35.62706082134951, 7790213), (35.13105720992483, 2519053), (33.76895447706966, 8445687), (31.530393861700702, 471681), (31.51624611119379, 6209396), (31.234843268722692, 5946388), (31.234843268722692, 892194), (29.872740535867518, 4226039), (29.872740535867518, 3798578), (29.872740535867518, 3293492)]}, {'query_id': '794429', 'scores': [(46.569008536102196, 1208450), (44.51291509151622, 1208449), (44.20148890909699, 64014), (42.141284933092805, 8618468), (42.141284933092805, 4652685), (42.141284933092805, 719934), (36.31505429909995, 6098420), (36.29910492195128, 1246185), (36.17933196901575, 3926405), (35.91957219536736, 4627633)]}, {'query_id': '801118', 'scores': [(47.140400328519995, 2932991), (46.2526073427832, 4799238), (46.11241858809377, 7615528), (46.11241858809377, 3975963), (44.660632527270224, 1930307), (44.497875184098184, 3263703), (43.84442590055729, 7486725), (43.35650768638846, 4214955), (43.34240916934306, 3360132), (43.3186959572832, 5219693)]}, {'query_id': '804066', 'scores': [(28.777996594823257, 73387), (28.464785772938193, 6152070), (26.953598122714453, 8567803), (26.953598122714453, 7922526), (26.953598122714453, 7621407), (26.953598122714453, 5655169), (26.953598122714453, 1504069), (26.953598122714453, 1259677), (26.286602248327824, 7427197), (26.286602248327824, 3298496)]}, {'query_id': '808400', 'scores': [(40.46393216142154, 3691917), (39.06400961808753, 3691923), (37.94796105262329, 3377142), (36.43127966878727, 7282136), (36.43127966878727, 7252539), (36.43127966878727, 4576228), (36.43127966878727, 1995351), (33.94461212838032, 5834423), (33.7462107843554, 5598346), (33.173143670917135, 5834421)]}, {'query_id': '809525', 'scores': [(50.17209296225508, 3793929), (45.049306736789724, 2077416), (44.03457266566295, 1724922), (43.08702265317709, 1428570), (42.54425791745811, 3793922), (42.54425791745811, 1428575), (41.529523846331344, 7059360), (41.529523846331344, 1428574), (41.529523846331344, 1428573), (41.45870925434113, 2077420)]}, {'query_id': '814183', 'scores': [(56.96149713957998, 8679359), (44.601420126516565, 8444060), (41.33399678576579, 8679357), (40.32149424919948, 7274881), (36.857312996525316, 510865), (36.857312996525316, 510863), (36.019646792058836, 510861), (34.4535751924041, 8552189), (34.4535751924041, 7013314), (34.4535751924041, 878335)]}, {'query_id': '819983', 'scores': [(28.598512620719063, 8145644), (28.598512620719063, 1338142), (28.499635414944272, 1617043), (28.43395625277155, 4580581), (28.431089247244948, 8312879), (28.42739108255476, 8328822), (28.375044959941107, 8439123), (28.36344605604693, 8831812), (28.36344605604693, 8045345), (28.361145313121714, 1221184)]}, {'query_id': '849550', 'scores': [(40.688357140999486, 1373164), (40.57660806039334, 7274031), (40.21694598955393, 6758594), (39.554534793403796, 8582102), (39.46437418890504, 7767168), (39.46437418890504, 4061418), (39.06422286267801, 8000555), (39.06422286267801, 7767169), (39.06422286267801, 1419091), (39.055003542353596, 5690724)]}, {'query_id': '85020', 'scores': [(56.88915261791594, 3152840), (56.52826372298955, 198481), (56.02350456410863, 198476), (55.59809281411971, 6290170), (54.99428617341411, 6974021), (54.70556586106912, 1770218), (53.35859334756081, 603541), (53.233173799482316, 147291), (53.031742404083516, 1640375), (52.98375191377768, 1777604)]}, {'query_id': '850358', 'scores': [(53.41830629231691, 2852825), (53.41830629231691, 2852824), (52.309587722334214, 2852830), (51.2972794627848, 6427506), (50.32041532552215, 2852827), (50.26001009482349, 2852826), (45.37836260169424, 8037013), (41.924703243395015, 389081), (41.858754195840305, 2278756), (41.85350544314281, 6427505)]}, {'query_id': '86606', 'scores': [(49.40269421509956, 5013458), (49.246146219700194, 8422274), (49.22244301165062, 6253490), (49.22244301165062, 6253489), (49.22244301165062, 677277), (49.012916106728454, 3077883), (48.83352820458131, 747160), (48.23535297371981, 6897895), (48.02723777980165, 677274), (48.02723777980165, 677272)]}, {'query_id': '877809', 'scores': [(55.869984403361535, 4888926), (55.47780792289071, 5940394), (55.47780792289071, 5940393), (55.06745011143402, 6704838), (54.721671557897515, 741714), (53.25889084621758, 4888929), (52.99297739881336, 7040022), (52.971660347486306, 7040017), (52.9187042974128, 1550610), (52.426733348915484, 7346771)]}, {'query_id': '883915', 'scores': [(58.50242039117779, 4042980), (57.068916864266875, 6172403), (56.36733001062542, 4042985), (56.27862100267711, 3535079), (55.98845923039123, 5406348), (55.93740933056204, 6172404), (55.035817020107885, 4857055), (54.73405602483984, 5559530), (54.659227889579704, 5406347), (54.61587722217311, 1845958)]}, {'query_id': '88495', 'scores': [(27.491116539453373, 1091557), (27.432907154040997, 2875829), (27.381860614867406, 6579243), (27.338157978312843, 6846565), (27.338157978312843, 1667678), (27.307487605962507, 7950301), (27.307487605962507, 5897696), (27.298933517708676, 5915611), (27.213329699178907, 5459872), (27.212738430234346, 3668359)]}, {'query_id': '911232', 'scores': [(89.68853793891248, 8816622), (85.41859984381088, 2999930), (83.84994531565226, 2999933), (81.29627474797786, 8816620), (78.24472828775302, 6928655), (76.2139999783883, 1879470), (72.17482094121382, 2999925), (69.34263652783989, 8816626), (68.59610124927131, 2999926), (67.46043502816724, 2999929)]}, {'query_id': '914916', 'scores': [(44.05395237044219, 6042350), (44.05395237044219, 2631739), (43.66548346270663, 2631743), (43.31529490436992, 7040507), (43.08485645055001, 8520449), (43.08485645055001, 8481901), (43.08485645055001, 5177082), (43.08485645055001, 2001628), (43.08485645055001, 2001626), (42.989938018266656, 6071696)]}, {'query_id': '91576', 'scores': [(39.56531196159263, 8279891), (38.88274507374275, 3780281), (38.74748942566475, 1382984), (37.15007528150846, 4161753), (31.1446280854899, 3540074), (30.77953416873299, 6805821), (30.77953416873299, 6137131), (30.77953416873299, 3255380), (30.096967280883113, 4066736), (30.096967280883113, 3880194)]}, {'query_id': '918162', 'scores': [(69.15507604954739, 2692389), (69.15507604954739, 2692384), (68.99332326678271, 2692388), (67.41761039320298, 267612), (66.21509124188817, 8217238), (64.81162371766727, 4658366), (64.81162371766727, 1290157), (64.81162371766727, 672942), (62.377876758993885, 2334496), (59.17634300056274, 1223048)]}, {'query_id': '938400', 'scores': [(47.320117404810816, 7239123), (45.09719990318791, 6499450), (45.09719990318791, 2969902), (41.5892501823983, 2870510), (41.110066410995756, 3074064), (39.89367683743545, 7029656), (39.89367683743545, 3424110), (39.89367683743545, 2371820), (39.89367683743545, 790219), (39.89367683743545, 47285)]}, {'query_id': '940547', 'scores': [(57.6578866672199, 4481817), (56.54169722074124, 6404591), (55.29245977260089, 4664999), (54.16604689920156, 3822464), (54.14978929217391, 4820847), (53.32637277018608, 7793152), (53.32637277018608, 4100051), (53.32637277018608, 3408094), (53.004557688150726, 1976277), (51.68505170960225, 2947470)]}, {'query_id': '945835', 'scores': [(52.36312382251422, 4431195), (50.867533754185395, 4539379), (50.668356091423504, 5548718), (50.668356091423504, 4431193), (50.24105694857643, 4425412), (50.24105694857643, 61200), (50.24105694857643, 61192), (49.727588705099855, 5548713), (49.727588705099855, 4539378), (49.17849199499683, 61193)]}, {'query_id': '978031', 'scores': [(35.57992070792969, 8744912), (35.048656349165086, 8744911), (34.25643386412537, 4755047), (34.23999172552556, 8744914), (33.222138367648235, 261672), (33.10307759611424, 8802560), (32.94621975425198, 3698979), (32.73017027395113, 5287819), (32.73017027395113, 4543384), (32.73017027395113, 2398979)]}, {'query_id': '985594', 'scores': [(36.03364863340221, 1657406), (35.53085353619195, 8677959), (35.53085353619195, 8677953), (35.53085353619195, 7740126), (35.53085353619195, 4286031), (35.53085353619195, 1657408), (35.53085353619195, 1657405), (35.53085353619195, 655890), (34.723334137642134, 7740128), (34.723334137642134, 7740125)]}, {'query_id': '99005', 'scores': [(82.82740211642795, 8257322), (81.26110407227809, 6493343), (79.35818902203503, 6493344), (79.03984766449662, 2875400), (79.03984766449662, 1375045), (76.64975190985699, 297028), (76.64975190985699, 297023), (71.74403287666426, 6327640), (71.56534267020356, 8257320), (70.84899092185182, 8476970)]}, {'query_id': '997622', 'scores': [(50.81806457480276, 4543865), (49.597337128496235, 73465), (47.44904533229087, 4518222), (47.44904533229087, 2275574), (47.03632455396879, 73459), (46.49856745710275, 4558331), (45.891441514696886, 4643397), (44.35027566089739, 8533344), (44.35027566089739, 3913148), (43.73199729370113, 5430569)]}, {'query_id': '999466', 'scores': [(40.5782020099497, 8748919), (40.5782020099497, 8748914), (37.93179753103994, 8748922), (37.93179753103994, 8748918), (33.55505166207379, 8748921), (33.55505166207379, 8748920), (33.55505166207379, 8748916), (33.55505166207379, 8748915), (33.55505166207379, 8748913)]}, {'query_id': '132622', 'scores': [(42.18282807056723, 5901873), (41.62670785372853, 962815), (40.40834526053401, 2459821), (40.40834526053401, 821354), (40.17642607512765, 2979805), (39.09006148784269, 7122663), (38.51185848962395, 5656719), (38.37160767508386, 3162996), (37.829143681034324, 6358747), (37.31557867780947, 3541644)]}]\n"
          ]
        }
      ],
      "source": [
        "print(query_processing(trec_dl_2020.queries_iter(), daat_bm25))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UIn-FsL5Hcdo",
        "outputId": "3800392f-506a-4c5b-96b1-6d8a59de7364"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "query_processing (145580.063 ms)\n",
            "[{'query_id': '1030303', 'scores': [(68.04124339861062, 8726436), (60.1427866714432, 8726435), (60.1427866714432, 8726433), (60.1427866714432, 8726429), (56.26487434885108, 8726437), (56.26487434885108, 8726434), (56.26487434885108, 8726430), (56.26487434885108, 7156982), (36.8093899826997, 1305521), (36.8093899826997, 1305520)]}, {'query_id': '1037496', 'scores': [(49.78270676861767, 7766587), (49.78270676861767, 7766585), (49.78270676861767, 5927420), (49.78270676861767, 4760912), (46.08250624801771, 4905511), (46.08250624801771, 4760914), (46.08250624801771, 4174377), (46.08250624801771, 3725937), (34.3057350592058, 4174376), (33.526059262405674, 3725932)]}, {'query_id': '1043135', 'scores': [(67.58094550401972, 8696961), (65.84778521905021, 650642), (64.07233051343995, 3378240), (63.811499082452244, 8696958), (62.87382325532231, 4994428), (61.23387105030339, 4355523), (61.23387105030339, 2514458), (60.077594538221234, 650641), (57.61450176736487, 2664995), (56.49291203767402, 4721484)]}, {'query_id': '1045109', 'scores': [(62.65581464672154, 8838896), (55.20688108779858, 8838890), (52.88217275039405, 8838893), (52.59407248914457, 8838892), (51.588943666145575, 8838891), (49.03746280776349, 8838899), (49.03746280776349, 8838898), (49.03746280776349, 8838897), (49.03746280776349, 8838895), (38.51262152953826, 7620329)]}, {'query_id': '1049519', 'scores': [(51.868481761168326, 7105), (50.76958755275562, 5355123), (46.32118326232374, 1316185), (45.971381551470046, 7106), (45.20014989004244, 7668125), (41.307972299785845, 5355121), (40.751745599610565, 6080565), (40.26732811546444, 5355126), (39.890437244812055, 6950445), (39.890437244812055, 6580258)]}, {'query_id': '1051399', 'scores': [(54.19868414964971, 4426188), (54.19868414964971, 69818), (54.19868414964971, 69813), (53.965490707744955, 2845499), (53.171797439158496, 7848674), (53.108276110252035, 3773237), (52.55272840666184, 8712317), (52.193951693351394, 2866972), (51.76370019693758, 3446937), (51.53270911799906, 3943228)]}, {'query_id': '1056416', 'scores': [(79.61916811336067, 8739209), (75.11227933549185, 2308626), (68.87502469490411, 8757245), (68.87502469490411, 2059786), (65.11806034305253, 3790242), (64.89493842990825, 895432), (63.01310460438273, 7321435), (62.78998269123845, 2348501), (62.78998269123845, 1367298), (59.07809351037385, 2348504)]}, {'query_id': '1064670', 'scores': [(60.32750960955309, 8680901), (58.767713457504655, 8680903), (58.085711433226464, 8680905), (56.959861539866935, 8680897), (54.612497156036284, 8680904), (52.37069897970967, 8680902), (45.323465075433155, 7734392), (43.08166689910654, 3807086), (41.95581700574701, 8337453), (41.95581700574701, 6878684)]}, {'query_id': '1065636', 'scores': [(58.78932722598091, 4371749), (57.178669194806446, 1062134), (57.02821144543303, 8490752), (49.19777543437664, 7411479), (48.34522226593835, 436583), (48.074107885396785, 3822992), (48.074107885396785, 597867), (47.806681760757506, 6307138), (47.32523509989382, 7411495), (47.18849018695526, 444919)]}, {'query_id': '1071750', 'scores': [(96.42909288222162, 3928913), (93.4766731950148, 8698268), (89.47034086348212, 8698276), (87.24605139920457, 8698274), (87.24605139920457, 8698269), (87.24605139920457, 3575934), (84.62555403932791, 8378390), (74.19949783794277, 8378386), (71.57900047806609, 8378389), (71.57900047806609, 8378385)]}, {'query_id': '1103153', 'scores': [(46.94883239928314, 8226438), (46.412659612159885, 8226436), (45.694343174868166, 8226445), (45.64339609448325, 7570096), (44.9960865467194, 6113506), (43.690650241919506, 8226444), (43.690650241919506, 8226440), (43.690650241919506, 8226439), (42.329596243837386, 8226443), (42.329596243837386, 8226441)]}, {'query_id': '1103791', 'scores': [(31.666238533045593, 8384424), (31.17049503301221, 8384416), (30.886013979015406, 4034845), (30.697285328434887, 8220282), (30.534630879918826, 4034852), (30.2830101983892, 1512326), (30.270988726409414, 8384417), (30.270988726409414, 8146212), (29.976348721672593, 8220278), (29.976348721672593, 8214165)]}, {'query_id': '1104501', 'scores': [(65.89412729708552, 6871582), (65.41840449297159, 6067096), (65.39582284977347, 5286080), (64.61878497960438, 2204057), (64.54957712134694, 2204054), (64.43256917565267, 2090898), (64.31747924474085, 5594847), (64.19704858360912, 7431269), (64.15606231274846, 2283306), (64.12101683648413, 3604439)]}, {'query_id': '1105792', 'scores': [(39.37059315121416, 8201616), (39.37059315121416, 8201611), (38.999172461108365, 8201618), (38.999172461108365, 8201612), (38.45499796165104, 8201613), (37.581020735249886, 8201615), (37.581020735249886, 8201610), (37.581020735249886, 6977017), (35.94706331197815, 8201614), (35.94706331197815, 7102720)]}, {'query_id': '1105860', 'scores': [(51.80051982448995, 3593906), (51.80051982448995, 3593905), (51.80051982448995, 3593904), (51.34894599516154, 8200940), (51.2784025316851, 8282567), (51.2784025316851, 8282566), (51.2784025316851, 8282565), (50.90628181279334, 7237676), (50.87358981321845, 3593907), (50.65799606338433, 4866505)]}, {'query_id': '1106928', 'scores': [(59.20863218150503, 4285175), (57.68234298179986, 2168463), (55.4006948609547, 5186636), (54.77346628597604, 7744803), (54.20077932621291, 8190627), (54.20077932621291, 8190624), (53.27899157588621, 2866124), (53.155024758113754, 8190620), (52.82548564081675, 1392371), (51.51175578482002, 6949062)]}, {'query_id': '1106979', 'scores': [(60.53563858569951, 8190140), (60.53563858569951, 8190137), (60.35317454410322, 7674419), (55.651224895562486, 7674423), (54.4033093271833, 5383176), (54.4033093271833, 4219185), (54.4033093271833, 1589841), (52.707424067591056, 8190135), (49.86026336047267, 8190134), (49.162371301808044, 348057)]}, {'query_id': '1107315', 'scores': [(39.54376615657153, 4598318), (37.51389793839698, 8187302), (36.022464891991085, 8817722), (36.022464891991085, 2696421), (36.022464891991085, 2696417), (36.022464891991085, 2696416), (35.22311638066542, 2687026), (35.22311638066542, 2687024), (35.22311638066542, 1708353), (34.15680819295444, 3430125)]}, {'query_id': '1107440', 'scores': [(61.89699856109981, 8186231), (58.33778121442552, 8186223), (45.6775153791329, 789989), (43.80313314161787, 1409567), (38.523388107855446, 3971063), (37.647486625284195, 1879424), (37.17878681008612, 8186227), (37.17878681008612, 5177729), (37.17878681008612, 5177724), (35.989959445628465, 2888123)]}, {'query_id': '1108450', 'scores': [(56.71887953761221, 1629571), (52.41776940719191, 4400204), (32.300004292406435, 1629577), (31.777632255332467, 6285212), (31.777632255332467, 6285210), (31.33422343316504, 8177145), (31.33422343316504, 4747082), (31.33422343316504, 4400207), (31.33422343316504, 4400205), (31.33422343316504, 1629573)]}, {'query_id': '1108466', 'scores': [(59.30142271208983, 8177012), (55.67270254789683, 4944265), (55.67270254789683, 3267306), (55.67270254789683, 753377), (55.199872139350035, 2702004), (54.82411557645084, 841054), (54.51524769260299, 5380573), (54.51524769260299, 3482588), (54.51524769260299, 3267303), (54.40273520904941, 2002688)]}, {'query_id': '1108473', 'scores': [(71.8535841426751, 8176948), (68.837321617921, 1188068), (67.88159874897809, 1188069), (64.05070212441933, 8176949), (56.883210440504, 1970473), (56.391214766044264, 3978700), (55.26259023758357, 8176947), (54.75703926520205, 1188063), (53.55061486545402, 5562098), (53.50519901171889, 357762)]}, {'query_id': '1108651', 'scores': [(52.042835175291344, 6348741), (51.707125565958556, 8175412), (48.84626280639219, 1371583), (47.09379244246787, 5027435), (43.784108385907516, 6985088), (43.784108385907516, 5997928), (43.29405003861879, 6663873), (42.9854927883285, 1325612), (42.88347701771074, 6985085), (42.88347701771074, 4617850)]}, {'query_id': '1108729', 'scores': [(63.10547382702718, 3405694), (57.22452254923036, 7789132), (53.69516866941264, 5449883), (48.865078065808376, 3405688), (46.03079622421798, 8174842), (44.74012911893431, 8322380), (44.46628101333716, 334958), (44.164593352223406, 7993178), (43.79687803700468, 1820798), (43.73131067345461, 5623921)]}, {'query_id': '1109699', 'scores': [(32.67479266338443, 2769074), (32.65745417997212, 4872045), (32.56902159014303, 8394502), (32.45251516227297, 7722707), (32.45251516227297, 7722706), (32.45251516227297, 7471054), (32.45251516227297, 2341119), (32.45251516227297, 737337), (32.30683047648765, 7744060), (32.30683047648765, 178437)]}, {'query_id': '1109707', 'scores': [(63.99305622763602, 1911564), (62.17738366690502, 1719374), (61.976129020952484, 5947139), (61.95931813768323, 4122615), (61.09866103592489, 771002), (60.89949286673625, 4122618), (60.87968502011701, 1719371), (60.53769115031774, 5740304), (60.192034498712474, 4285843), (59.92002711015108, 4099609)]}, {'query_id': '1109850', 'scores': [(60.34972932491009, 8164028), (58.28235902800127, 8164035), (58.28235902800127, 8164030), (56.74119317420177, 8164033), (46.99295859198112, 3633621), (46.99295859198112, 3633617), (42.70642938809392, 8164032), (40.5782020099497, 8164036), (33.55505166207379, 8164029), (28.039054851875697, 5382247)]}, {'query_id': '1110678', 'scores': [(47.78458294762738, 7139329), (45.84152116584514, 8155914), (45.84152116584514, 6632692), (45.84152116584514, 2342590), (45.84152116584514, 2342589), (44.97464403661556, 8155909), (44.97464403661556, 5719597), (42.62799591135909, 2714688), (42.62799591135909, 2126363), (42.62799591135909, 2096160)]}, {'query_id': '1112142', 'scores': [(81.1818694630267, 2542782), (80.2780492424212, 3651292), (80.2780492424212, 3651291), (80.2780492424212, 3651290), (79.8570649461349, 7866090), (72.65079473401737, 2238202), (69.50557890927026, 1946438), (68.23903268901505, 8240097), (68.23903268901505, 7844243), (68.23903268901505, 5670185)]}, {'query_id': '1113042', 'scores': [(50.769852428627246, 7072957), (50.36054095803586, 2410652), (49.2616863479491, 355186), (48.33019169852579, 2499270), (47.863811684552275, 3521102), (46.99188383234091, 7834150), (46.99188383234091, 4931496), (46.99188383234091, 4243566), (46.454216943749486, 2326525), (46.454216943749486, 2326519)]}, {'query_id': '1113256', 'scores': [(92.31619375511241, 7561115), (91.67121386472638, 8130535), (91.67121386472638, 7561120), (89.3871919777977, 7561117), (87.40058143324417, 7561116), (86.7519549223406, 7561114), (86.05712136539144, 8130530), (86.05712136539144, 7561113), (83.42553093045187, 7561111), (76.74533982637449, 8130528)]}, {'query_id': '1114166', 'scores': [(43.684903129639494, 6019023), (43.24267516869766, 3594677), (43.24267516869766, 3142379), (43.24267516869766, 2463783), (42.100667253468956, 8478109), (41.64177063981604, 5989751), (41.372197942512756, 1816877), (41.365037944904195, 5193359), (40.86959688946432, 5005484), (40.86959688946432, 3951381)]}, {'query_id': '1114286', 'scores': [(26.74043873052966, 6846966), (26.74043873052966, 4956824), (26.571946729577576, 2589532), (26.46978385632458, 4956825), (26.02173778202728, 8720406), (26.02173778202728, 3960437), (26.02173778202728, 2177460), (25.944776379603326, 8413644), (25.944776379603326, 6040011), (25.86209312148852, 6269131)]}, {'query_id': '1114993', 'scores': [(31.87074657982275, 4104358), (30.981149493221995, 8113957), (30.467171891895834, 6612208), (30.467171891895834, 6203023), (30.467171891895834, 6011208), (30.467171891895834, 4956693), (30.467171891895834, 3271410), (30.467171891895834, 2086636), (30.467171891895834, 1881796), (30.467171891895834, 1724571)]}, {'query_id': '1115210', 'scores': [(56.23960159264233, 8111887), (51.838241468000774, 8111878), (45.85690591400068, 8111885), (45.85690591400068, 8111884), (45.85690591400068, 8111882), (32.99891697009785, 8390254), (32.99891697009785, 7440361), (32.23150029637465, 7440364), (32.23150029637465, 7440363), (31.498966198729775, 7440366)]}, {'query_id': '1116380', 'scores': [(51.55126145420928, 8100548), (51.427279378454784, 8100546), (51.427279378454784, 8100543), (40.805316694675206, 521341), (36.097010922212675, 4118339), (36.097010922212675, 1422757), (36.097010922212675, 137869), (30.22340281054382, 5104772), (30.22340281054382, 1645711), (30.22340281054382, 444969)]}, {'query_id': '1117817', 'scores': [(44.85787012481133, 5310910), (44.85787012481133, 3516296), (42.20441204717148, 4486487), (40.19114717744313, 7531176), (36.53236138663283, 6537918), (36.27688342434477, 1005145), (35.48705971968379, 1035058), (35.203186468620046, 109686), (34.67246691228346, 5970406), (34.67246691228346, 5447234)]}, {'query_id': '1117886', 'scores': [(45.155571001571154, 2921720), (43.84248561201732, 556178), (43.51556748799402, 8456607), (41.77729167614943, 7460949), (41.77729167614943, 7460948), (41.77729167614943, 7024723), (41.77729167614943, 5376945), (41.61933775281216, 5094343), (41.405913909284045, 5301093), (40.55085061517856, 8407176)]}, {'query_id': '1118370', 'scores': [(28.23447351263131, 8656398), (27.84183144059962, 7476349), (27.84183144059962, 2951456), (27.573872754238188, 2224024), (27.536156936053047, 8081645), (27.536156936053047, 2224022), (27.143514864021355, 6211291), (27.143514864021355, 3608410), (27.143514864021355, 1284328), (26.731574793473737, 861060)]}, {'query_id': '1118426', 'scores': [(42.41389325551132, 8081194), (42.41389325551132, 8081193), (36.39603637919815, 8081199), (36.39603637919815, 8081196), (36.39603637919815, 2424347), (34.77247446904838, 6622100), (34.77247446904838, 5598458), (34.77247446904838, 2424353), (32.80075099810084, 6208983), (32.26026676652181, 2469356)]}, {'query_id': '1119118', 'scores': [(47.24855375761641, 8074735), (46.37104402641371, 8074733), (39.56927285253072, 6026130), (39.56927285253072, 5472178), (39.56927285253072, 3609036), (37.732918441716485, 8074741), (37.732918441716485, 8074734), (31.927854066067788, 5451461), (31.927854066067788, 5185340), (31.927854066067788, 4246892)]}, {'query_id': '1119543', 'scores': [(73.14390257979977, 8070706), (67.26154423079493, 8070713), (63.71001486756422, 1603603), (63.49580937728193, 405831), (62.682088206903956, 405832), (61.90666894198526, 5052354), (61.678958097108485, 8070707), (58.42565670222966, 4770699), (57.7901822467058, 3612262), (50.40572856579277, 3612261)]}, {'query_id': '1120588', 'scores': [(50.74883407765331, 8060866), (50.034889642520575, 8060864), (49.07511407369297, 8060867), (47.98424289062399, 8060861), (46.949503514089606, 8060862), (46.65909800121558, 8060868), (44.18036897245296, 8060859), (43.48233885670176, 8060863), (43.48233885670176, 8060860), (42.447599480167376, 2313348)]}, {'query_id': '1121353', 'scores': [(57.75758307518821, 8053113), (56.43735242720638, 8053117), (48.66050593121671, 1854970), (45.295535427894414, 4872676), (44.20432528727399, 4210422), (44.20432528727399, 623601), (41.99125020017842, 6379159), (41.31466705546898, 5606064), (40.477130174718546, 5277905), (40.477130174718546, 937044)]}, {'query_id': '1121879', 'scores': [(63.179294306621756, 2603303), (63.16796157074404, 3024534), (61.838159871381336, 1307737), (61.506273146994516, 8048097), (61.506273146994516, 2603307), (61.506273146994516, 1406295), (57.133980265637504, 2193862), (57.02264321039648, 4995632), (56.92733381460421, 8048100), (55.609269659927776, 1025282)]}, {'query_id': '1122138', 'scores': [(27.919834320303984, 4342247), (27.8389152606343, 4342246), (27.79093190871536, 5871298), (26.811595024827888, 5463811), (26.519075572290216, 6283517), (26.25681570236712, 7067825), (26.25681570236712, 6963235), (25.766503771641332, 3967084), (25.766503771641332, 3647404), (25.766503771641332, 3345300)]}, {'query_id': '1122767', 'scores': [(55.379751805473646, 7544609), (55.379751805473646, 1575377), (54.0375198358854, 8039590), (54.0375198358854, 5877496), (52.61612157586279, 5877500), (50.630315605392155, 6851717), (50.630315605392155, 6179207), (50.630315605392155, 2859907), (50.0003116306419, 2019371), (48.08165371127525, 4550504)]}, {'query_id': '1122843', 'scores': [(77.38656103218096, 2332417), (77.360329323602, 2332424), (75.96491948664827, 8772747), (75.48098511711144, 225969), (73.60972450880568, 7998359), (72.72505845313061, 4947308), (72.43026526635228, 2332416), (71.94918042370892, 3880592), (71.85814009539307, 2332425), (71.58676217309764, 3312892)]}, {'query_id': '1123657', 'scores': [(63.052005507112, 8032346), (50.03639026539902, 8032342), (47.47299756079464, 8032349), (45.27116795141025, 7221025), (44.69577592931092, 5571459), (44.262960619391436, 8032348), (44.262960619391436, 8032347), (44.262960619391436, 8032343), (44.10126654142709, 7988749), (44.06405784924217, 5572500)]}, {'query_id': '1124552', 'scores': [(55.44700893803659, 5782491), (46.36459394400007, 3442729), (44.06691367266147, 4743535), (44.06691367266147, 3442725), (42.75821015325354, 7082651), (42.642721094215034, 8214271), (42.55223627018191, 2601066), (42.252142132792564, 2601070), (42.252142132792564, 1797406), (41.691096571586385, 6263843)]}, {'query_id': '1125632', 'scores': [(63.47869767216254, 7011199), (63.32635300623171, 6721396), (63.078436368782235, 1278069), (62.82611482802085, 7191566), (62.82611482802085, 1475286), (62.82611482802085, 1243662), (62.68867946320094, 6934400), (62.68867946320094, 6543904), (62.041313061301025, 7011204), (62.041313061301025, 5847127)]}, {'query_id': '1125755', 'scores': [(32.17472862571786, 1744931), (30.980582314378243, 7420642), (30.980582314378243, 2206347), (30.276862487099784, 7856574), (30.276862487099784, 3808440), (29.98851592373698, 7254544), (29.08271617576017, 7387332), (29.08271617576017, 7254547), (29.08271617576017, 2938068), (29.08271617576017, 482655)]}, {'query_id': '1126523', 'scores': [(43.40952262665484, 8805840), (42.215376315315225, 8805844), (42.215376315315225, 8805841), (38.40073155434851, 8805837), (33.06025459504485, 8805846), (33.06025459504485, 8556233), (33.06025459504485, 3214173), (33.06025459504485, 3025043), (33.06025459504485, 2841635), (29.245609834078138, 2774244)]}, {'query_id': '1126738', 'scores': [(73.44858039702677, 8790091), (73.44858039702677, 1091656), (71.97337726278052, 1091662), (71.97337726278052, 1091658), (66.09775988925377, 8790089), (66.09775988925377, 1091654), (64.69687693845499, 8790093), (64.69687693845499, 8790090), (64.69687693845499, 8790088), (62.64053317556626, 8790087)]}, {'query_id': '1127004', 'scores': [(50.772228502850425, 6481277), (50.50644958893912, 6531608), (50.34667475165446, 695631), (50.343155707666256, 5224906), (50.08089583774316, 1756214), (49.9770828639771, 3961675), (49.85636282092867, 6011954), (49.714822994054, 860288), (49.60403746263563, 6654716), (49.60403746263563, 4964550)]}, {'query_id': '1127233', 'scores': [(33.43839401182032, 8100081), (33.12034911178168, 8100077), (32.94702305514685, 8527129), (32.79347511389552, 5400207), (32.67990674569691, 6611628), (32.565921152783524, 124809), (32.47666828286078, 6438836), (32.47666828286078, 3842712), (32.47666828286078, 124808), (32.35852063029373, 7619778)]}, {'query_id': '1127540', 'scores': [(36.59702052801311, 8726963), (36.59702052801311, 8726962), (36.59702052801311, 186476), (35.01607652466533, 8726961), (35.01607652466533, 8726956), (35.01607652466533, 8726954), (32.732419360013246, 2574460), (32.732419360013246, 2044372), (32.732419360013246, 2044369), (32.732419360013246, 2044368)]}, {'query_id': '1128456', 'scores': [(67.20456401098838, 8649622), (57.76436548388226, 1459777), (50.76898220701672, 7526299), (49.37914260305433, 4084616), (48.303610569315055, 8649615), (48.303610569315055, 3721820), (45.85108822376522, 3121928), (45.85108822376522, 3121927), (45.85108822376522, 2534794), (45.85108822376522, 2516294)]}, {'query_id': '1129081', 'scores': [(37.17436875892819, 8599986), (32.885018517513394, 7871513), (32.885018517513394, 6962228), (32.885018517513394, 6794362), (32.885018517513394, 4178203), (32.885018517513394, 3762080), (32.885018517513394, 2770798), (32.885018517513394, 2156392), (32.885018517513394, 1044198), (32.885018517513394, 687772)]}, {'query_id': '1130705', 'scores': [(29.689061772055993, 1468216), (29.54307887055976, 8442374), (29.54307887055976, 1621163), (29.440090933202825, 6762105), (29.440090933202825, 6762102), (29.30440841255797, 6628960), (29.30440841255797, 6233460), (29.30440841255797, 5379660), (29.30440841255797, 3570934), (29.117524940726383, 1910013)]}, {'query_id': '1130734', 'scores': [(63.84021675814829, 2546803), (62.24989648942528, 7212169), (61.22288170736921, 8460898), (61.107585650625595, 8460901), (60.75301540282305, 8460902), (56.01056272100301, 4311761), (56.01056272100301, 2810717), (56.01056272100301, 1081476), (55.53790282604993, 3823605), (54.65571400274294, 7427232)]}, {'query_id': '1130847', 'scores': [(67.05464563572143, 5474954), (63.94480815670349, 4201753), (62.16186061328391, 7332266), (62.16186061328391, 7332262), (61.340966056268755, 7332261), (61.1328327729, 6209899), (60.864310037527815, 8299223), (60.23497606629693, 8299227), (60.118149081715956, 1458056), (58.931600446827275, 6149589)]}, {'query_id': '1131069', 'scores': [(55.15551722305412, 8432048), (54.24704755495573, 8432050), (52.14839993614961, 8191743), (52.14839993614961, 1780971), (52.14839993614961, 1780963), (45.691755539423795, 6907802), (45.13610475303173, 1780966), (44.2794966449483, 6809701), (44.2794966449483, 6809698), (43.90531916252551, 6809699)]}, {'query_id': '1132044', 'scores': [(52.26632639511176, 5858455), (51.334243825644954, 1014774), (50.643211669671665, 5858453), (50.57880414504043, 2121532), (48.08817157388742, 7645045), (47.79800040884696, 1691946), (46.306366919840315, 1818480), (46.14078024084789, 6166188), (45.98443939626003, 1650397), (45.79050502439325, 5810405)]}, {'query_id': '1132247', 'scores': [(72.1955852790043, 6246993), (68.80605331931017, 1210363), (68.80605331931017, 1210358), (68.80605331931017, 1210357), (66.74093641141356, 7390343), (65.3982141842798, 7390342), (65.3982141842798, 6525827), (65.39718289242735, 8238483), (64.6102215584632, 4623273), (63.953129242230176, 7390347)]}, {'query_id': '1132532', 'scores': [(59.60401572783728, 23966), (56.61600785589939, 6994907), (52.94313594278585, 4893092), (51.99571830273818, 6832793), (51.98341929772242, 8374347), (51.78606666869286, 3299150), (51.78606666869286, 2762578), (51.68063277226257, 5989422), (50.82947348367798, 1976995), (50.13695259787546, 6832795)]}, {'query_id': '1132842', 'scores': [(60.856756741885064, 3158781), (58.39076225103622, 7364932), (57.95182109155522, 834533), (57.3113316585592, 6536692), (57.22424214109628, 6570513), (56.85987297446198, 4874198), (56.75267792105353, 4132635), (56.533557076033645, 2324752), (56.30999901697635, 2941398), (53.63309923461685, 2391268)]}, {'query_id': '1132943', 'scores': [(48.424366336331346, 423711), (47.855090450069376, 5552077), (47.80704214079758, 6253953), (45.54683476242134, 622662), (45.247760326156964, 3437105), (40.689922629440574, 6469809), (40.262600755389386, 5782177), (40.13237429703533, 2107961), (40.124596856585875, 2248898), (40.124596856585875, 2107962)]}, {'query_id': '1132950', 'scores': [(40.64063818226049, 8274826), (39.48926366135125, 8619165), (38.954800117006194, 5867940), (37.83142796540978, 8274822), (37.83142796540978, 2303442), (37.39777270313935, 2132514), (37.39777270313935, 261777), (37.353298383953, 5845390), (36.58693749011307, 6590922), (36.2792718712176, 6625752)]}, {'query_id': '1133485', 'scores': [(38.68264716789245, 5221748), (38.53758164410002, 8330714), (38.51171756975086, 3156351), (38.474855652657965, 3183897), (38.45358261098119, 2418248), (38.42309613271226, 6195711), (38.38805974627046, 8692792), (38.34727342929999, 5630061), (38.33293869903945, 4090344), (38.144682510779624, 3089617)]}, {'query_id': '1133579', 'scores': [(46.19416112309672, 8220878), (45.67825283033753, 4931644), (44.44967034015408, 1817055), (43.48514614064777, 4931642), (43.48514614064777, 4931641), (41.0367385572856, 670206), (39.952171119483616, 8220879), (39.952171119483616, 5335041), (39.952171119483616, 3691486), (39.952171119483616, 2448461)]}, {'query_id': '1134094', 'scores': [(43.76191857295562, 8172161), (41.25123084186529, 8172166), (41.25123084186529, 7182375), (39.90516798705288, 6426847), (39.90516798705288, 6426846), (39.90516798705288, 3445457), (38.64233377456488, 8172159), (36.48823920175982, 7182378), (36.48823920175982, 6426844), (36.48823920175982, 5357491)]}, {'query_id': '1134207', 'scores': [(27.84220368943989, 1698802), (27.777080376791655, 374550), (27.371782415275803, 7155676), (26.98490708837431, 5592724), (26.792907708165075, 8817805), (26.792907708165075, 853685), (26.732039707896472, 3225711), (26.702302643158767, 7049025), (26.418520741440894, 8246744), (26.26161843373238, 3362951)]}, {'query_id': '1134431', 'scores': [(60.329909957113465, 3128065), (58.98846276501494, 8141222), (57.99616841203113, 8141223), (56.646980063516, 8141221), (56.578934153071366, 7974008), (54.571988313836954, 8141220), (50.512023948032635, 5010274), (50.386254816336695, 2339181), (50.24871688092816, 6720464), (50.24871688092816, 3243219)]}, {'query_id': '1134680', 'scores': [(81.37645784571794, 8118937), (70.29330432308248, 8118936), (67.93812033206164, 8118930), (57.90424061179279, 8796589), (57.46690544186423, 8307236), (57.072900122153335, 2456858), (57.072900122153335, 2200409), (57.072900122153335, 1865327), (57.072900122153335, 1865324), (57.072900122153335, 237951)]}, {'query_id': '1134939', 'scores': [(42.853951301835025, 322652), (42.82268562839048, 8093333), (41.41079790679133, 8093330), (36.63262891754617, 6204714), (36.63262891754617, 2090588), (33.22861335484515, 8455233), (33.22861335484515, 8455232), (33.22861335484515, 7543137), (32.47341759678049, 8093327), (32.47341759678049, 7543128)]}, {'query_id': '1134988', 'scores': [(80.10000590743692, 8088407), (77.98896099062691, 8088409), (74.36393657919554, 380755), (73.71837614401221, 8088404), (70.0404375827346, 380753), (70.0404375827346, 380752), (69.86915343009824, 7296294), (69.6446537262785, 380748), (69.00493401986287, 6432242), (67.9293926659246, 380749)]}, {'query_id': '1135268', 'scores': [(43.49021735717509, 8386270), (43.21097059091062, 2966472), (42.93358408437036, 5717144), (42.906508451832195, 6259848), (42.906508451832195, 6087160), (42.42816725484123, 4260), (42.373556842606064, 5225950), (42.373556842606064, 1683963), (42.211323044619604, 8371436), (42.033828410392914, 3622462)]}, {'query_id': '1135283', 'scores': [(35.10163189483498, 8061944), (32.67701976166231, 2582837), (31.993737003931642, 4502908), (31.517136554906347, 3368379), (31.381455976189663, 7770955), (31.381455976189663, 7043720), (31.381455976189663, 6979076), (31.381455976189663, 5959072), (31.381455976189663, 5834915), (31.381455976189663, 3989820)]}, {'query_id': '1135413', 'scores': [(69.42058685825441, 8050662), (66.59143012957217, 8050667), (60.01578563077492, 8050665), (59.707485362123684, 2115732), (58.229292786921036, 6672865), (57.9222117134223, 7522683), (57.55417662868983, 6039815), (57.55417662868983, 5287000), (57.55417662868983, 2061136), (57.45661105415273, 5118096)]}, {'query_id': '1135626', 'scores': [(51.93404238517209, 2003615), (51.62224336752075, 4035834), (51.580530116613886, 7676917), (51.55423480995148, 6532689), (51.27003752001791, 2902886), (50.29887206225635, 4619335), (49.39151033932774, 4892864), (48.74735233333061, 7714006), (48.64615955999031, 4079092), (48.43538885440741, 4035841)]}, {'query_id': '1136043', 'scores': [(48.49227419794402, 8728069), (48.426503167881336, 8728071), (46.5246059242415, 8728070), (46.49675513529931, 8728072), (45.50936051747823, 1056540), (44.248514810669484, 8728076), (44.145425304066265, 1056534), (43.471983580163986, 7018326), (43.471983580163986, 5326197), (42.89524230080494, 8728077)]}, {'query_id': '1136047', 'scores': [(58.98542692069104, 8724031), (56.61585013690356, 8724032), (55.22938011688501, 3988779), (54.35552697979545, 3597758), (54.35552697979545, 3006066), (53.232541218182696, 7009393), (52.08997302746144, 8328670), (51.82862348204492, 512482), (51.49548411356448, 2815435), (50.82993629738824, 8724036)]}, {'query_id': '1136769', 'scores': [(75.39595534457717, 8082043), (74.15151279910532, 189017), (72.32974381400743, 129247), (71.31048175331118, 189019), (70.16068402993051, 8082050), (70.152494218761, 4713242), (69.19259156812069, 189016), (67.21298740028747, 8082046), (67.21298740028747, 5718075), (63.81640879032942, 5508127)]}, {'query_id': '1136962', 'scores': [(112.88596394931662, 7631079), (112.75965569938819, 6185710), (111.92902232390603, 6185711), (111.30390009177441, 6185717), (110.63352155344846, 8336574), (109.71769576945059, 6185714), (109.71769576945059, 6185712), (103.55176921943426, 596001), (100.19062985407479, 6185716), (99.81769162303566, 4601613)]}, {'query_id': '118440', 'scores': [(38.08312964079179, 7035722), (38.08312964079179, 1569786), (36.56341517398157, 8319567), (35.918473572157644, 4737856), (35.918473572157644, 1569787), (34.7354464060783, 8412702), (32.1592702708219, 6750552), (32.1592702708219, 6750548), (31.476644250922377, 6750547), (31.476644250922377, 2975214)]}, {'query_id': '119821', 'scores': [(42.15927546938831, 7093228), (39.9736450641465, 7172227), (39.9736450641465, 6527223), (38.552187189210215, 8012784), (38.552187189210215, 6267963), (38.552187189210215, 3365586), (38.552187189210215, 259724), (33.44246188444183, 8012780), (32.682405932522705, 2212944), (31.261431761543456, 2212940)]}, {'query_id': '121171', 'scores': [(39.900354287263255, 8260192), (34.54604464183217, 7165126), (34.54604464183217, 6930941), (34.54604464183217, 2337570), (29.631480688369646, 451171), (29.479264177984188, 8643275), (29.479264177984188, 451173), (29.002510579418402, 2550811), (29.002510579418402, 451172), (29.002510579418402, 259437)]}, {'query_id': '125659', 'scores': [(43.57127492663355, 6976559), (43.57127492663355, 4522348), (43.57127492663355, 1335471), (42.12750378945115, 829647), (41.81379309610916, 5881859), (41.245034769340336, 8290885), (41.245034769340336, 6635625), (41.245034769340336, 2745708), (41.245034769340336, 1106134), (39.80126363215793, 5693282)]}, {'query_id': '135802', 'scores': [(44.30502903866332, 8692623), (43.110882727323705, 8692626), (42.82869160357404, 8692628), (41.63454529223443, 8692619), (41.63454529223443, 6037338), (41.63454529223443, 6037331), (37.88691949546934, 8731214), (37.88691949546934, 7327478), (37.88691949546934, 5394718), (35.816350651275435, 8692624)]}, {'query_id': '141630', 'scores': [(74.23435832367451, 2473236), (70.95430172369446, 8319991), (70.79139859818275, 6290795), (69.17287093978382, 8319989), (68.7532413262578, 7985119), (68.45988473421217, 2366121), (68.23941274199179, 8319987), (68.07681391968251, 3500851), (67.75577831700812, 3500859), (67.38377377919639, 6496223)]}, {'query_id': '144862', 'scores': [(47.03296268584424, 92239), (46.85451620671749, 92237), (46.41977710013721, 92238), (46.303802544594035, 2865950), (45.841647474804866, 92233), (45.04799952283608, 5292762), (44.91752322502883, 6150954), (44.57574445755628, 8320087), (44.57574445755628, 7245936), (44.57574445755628, 92241)]}, {'query_id': '156498', 'scores': [(61.9888765320714, 4273410), (53.085465718926976, 5606985), (52.187491504131046, 8823842), (51.484728624873746, 3288004), (51.18236403261821, 5043014), (50.99490726964783, 7075146), (50.99490726964783, 3397582), (50.7483870606257, 3397578), (50.68386759140607, 3301559), (50.47597954773579, 3288005)]}, {'query_id': '166046', 'scores': [(56.70604024220032, 7947463), (56.70604024220032, 1076792), (56.70604024220032, 1076791), (55.3531630803186, 1076793), (51.515912760751235, 1076795), (51.515912760751235, 1076794), (46.37415447435012, 96920), (45.87658876205094, 7947466), (44.98110483922546, 96927), (44.64407493868207, 7865285)]}, {'query_id': '169208', 'scores': [(46.1293108649256, 4859433), (45.25272675187287, 4859432), (45.17568359612078, 4580137), (45.044664136325935, 4762140), (44.6702035085016, 7251639), (44.451314500235696, 4202275), (44.33368093653263, 1395417), (44.15687115058914, 85987), (43.920419240465975, 2888366), (43.91192172575532, 4762142)]}, {'query_id': '174463', 'scores': [(44.757460801514235, 8304456), (44.14240235360076, 4501665), (42.6438588329258, 102528), (39.75116285018439, 6294949), (39.567975704697155, 7089630), (39.567975704697155, 7089629), (38.740159850529935, 8304458), (38.420287007787394, 7402420), (36.520498788441714, 7611683), (36.520498788441714, 5793989)]}, {'query_id': '175920', 'scores': [(91.71097317824558, 8234235), (80.44016322758429, 101345), (71.7350906626466, 8537849), (71.55857417490498, 7615731), (71.55857417490498, 5673665), (70.68012422444055, 8537850), (69.96805339615102, 7360509), (69.96805339615102, 6815325), (69.96805339615102, 5717848), (69.96805339615102, 5673664)]}, {'query_id': '177604', 'scores': [(49.127188538359235, 8451995), (47.6872321025401, 8451992), (43.262039574839875, 5627757), (43.23395161262104, 2226834), (43.01994243804827, 5034647), (41.636537468464745, 3444466), (41.636537468464745, 1308162), (39.29998151875854, 8451989), (39.08671906601341, 8451987), (38.74797180084217, 8451991)]}, {'query_id': '181626', 'scores': [(40.53255333688902, 43641), (39.81660366233994, 436351), (39.423961590308245, 7795335), (39.423961590308245, 4798089), (39.423961590308245, 2183331), (39.14320446662304, 71074), (38.68810690588239, 8069297), (38.478089053400325, 71073), (38.42725479207395, 5427354), (38.42725479207395, 4091013)]}, {'query_id': '197312', 'scores': [(41.162546692577244, 2317108), (41.162546692577244, 2317106), (40.70629139945948, 5823013), (40.01806817887634, 5823008), (39.95085483146295, 1179432), (39.58592337529094, 5823015), (39.58592337529094, 1179433), (39.53481117066497, 1665689), (39.32952437503172, 2317100), (39.32952437503172, 1239571)]}, {'query_id': '206106', 'scores': [(59.081627578287424, 6479522), (57.650528259341556, 75400), (57.06990662658893, 7079997), (56.49522468802833, 1390597), (56.232731249131675, 5093731), (55.87373213949044, 8427974), (55.6264222680681, 8737574), (55.43360929792247, 3217609), (55.43360929792247, 2947646), (55.00121408359372, 7118029)]}, {'query_id': '227873', 'scores': [(58.34724123600914, 7276796), (57.03242917058785, 8453473), (55.9009242815982, 7276798), (55.9009242815982, 3250369), (55.9009242815982, 2054488), (52.62294352267836, 7019128), (51.96420519159956, 3324178), (47.627275454116344, 1900788), (45.111732400332265, 8453475), (44.89234565470312, 8453476)]}, {'query_id': '23849', 'scores': [(41.68579344085422, 4834498), (40.61391470104821, 2674124), (40.4297704726566, 6728787), (40.34534166780886, 4348282), (40.229052936603495, 5888570), (39.78504141745572, 2373852), (39.31529238166785, 641196), (39.27346292800285, 8246990), (39.27346292800285, 2017213), (39.02916558598355, 5363400)]}, {'query_id': '240158', 'scores': [(58.09637454065858, 2043788), (56.44957479929517, 991184), (56.413675929163745, 6536692), (55.79646497250479, 1567595), (55.35762668220193, 4874198), (51.10190407357834, 2819781), (51.01242314410505, 2819779), (50.22028401002015, 3542898), (49.67444386045439, 2819784), (49.615693447155564, 6239283)]}, {'query_id': '245052', 'scores': [(54.78227259196777, 3828386), (48.75521741769647, 5023796), (47.995572749581626, 5878365), (47.975278823529166, 4205737), (47.975278823529166, 66616), (47.56453582649563, 8282910), (47.524744090736924, 5666135), (47.34034464993742, 5143412), (47.197713435129714, 8279354), (47.197713435129714, 3379257)]}, {'query_id': '246883', 'scores': [(58.49862076925101, 3050794), (57.60830943726493, 8699192), (55.91784080624278, 5050393), (53.65520307780655, 8699195), (52.6072760439256, 4959448), (51.86877037029821, 5683913), (51.86877037029821, 2413876), (51.444031264185185, 6242496), (51.11560305782305, 6083923), (51.02702559511082, 8420183)]}, {'query_id': '253749', 'scores': [(45.057801432053665, 6190136), (44.08063422763945, 3516374), (43.65190932842373, 4070966), (40.541906325052345, 3516370), (39.958790713231274, 3516368), (38.512951418593566, 8254833), (37.06755826560743, 8254894), (35.74893113763399, 8521273), (35.70780947966956, 6260689), (35.11229189839443, 6447304)]}, {'query_id': '256942', 'scores': [(61.19538851256883, 2602557), (51.801731127628024, 2364119), (51.03897487994514, 7213042), (50.680085431483825, 4890686), (50.28471879180756, 8327014), (50.22386515983437, 3262199), (49.96352603379433, 5297913), (49.69133710998485, 5083782), (49.69133710998485, 4604331), (49.69133710998485, 1183278)]}, {'query_id': '257119', 'scores': [(55.75933340374534, 8499023), (54.39758593496521, 4363515), (50.102035098427194, 4363509), (47.28987635244748, 8032136), (46.26756220282472, 3818024), (46.14063039509142, 889398), (45.75958434457088, 8276379), (45.58051708303541, 5853762), (45.572972626528454, 2112755), (45.54031879200133, 4212490)]}, {'query_id': '258062', 'scores': [(66.69137483880327, 8694161), (66.27035011023386, 4159145), (65.09502443451943, 18103), (63.328158808857125, 3683402), (61.77618395543402, 7918982), (61.21532207399008, 2707010), (61.103314797780655, 4174313), (60.30109614516119, 3195767), (59.61420479423781, 91493), (59.26003812686205, 4174307)]}, {'query_id': '26703', 'scores': [(37.324271411427155, 3532417), (35.531673443307156, 3532415), (35.531673443307156, 1485077), (30.284021724135023, 380356), (29.30558654255803, 7265225), (29.30558654255803, 1236259), (29.22874653709181, 3733548), (28.68822632325314, 3921274), (28.599411585419848, 3132961), (28.599411585419848, 524170)]}, {'query_id': '273695', 'scores': [(57.39333772315531, 8282608), (56.294011226722446, 5542707), (55.4143555432112, 7381998), (55.303084413960335, 5542705), (54.76567460544736, 2714742), (54.26834503742595, 7381994), (53.681698564261424, 5719948), (53.681698564261424, 4171983), (53.681698564261424, 413461), (53.6196640996621, 413460)]}, {'query_id': '302846', 'scores': [(67.08395635280485, 8418433), (64.9607030889068, 8418435), (61.47596110236128, 447099), (61.04943906015667, 8283558), (60.91228638190829, 6263014), (60.76169925588102, 3892531), (60.71801612755096, 3566357), (60.531106154491084, 5696466), (60.23256546751837, 447101), (59.93179017426158, 4234633)]}, {'query_id': '318362', 'scores': [(53.40244884319692, 5654804), (52.25139353733929, 3145979), (52.1992268837149, 429301), (50.640570936361115, 4299889), (49.86932284569948, 1714531), (48.99399023990718, 1714527), (46.9609011023055, 4299887), (46.83577965533793, 543937), (46.10061642160003, 519111), (45.40099425002858, 3145978)]}, {'query_id': '324585', 'scores': [(56.6735402484278, 1051441), (51.95359889081024, 8290314), (49.76506995256631, 8290310), (48.9037615977678, 3608325), (47.650925464522864, 7977807), (47.158096835675565, 7941125), (45.060602352130545, 7359510), (44.83921786558659, 8310496), (44.67331676706778, 405282), (44.44231889273502, 2194313)]}, {'query_id': '330501', 'scores': [(52.957923034839915, 6006326), (52.02650095566991, 3718639), (52.02650095566991, 1750071), (48.39151989218534, 6647777), (48.17758414622325, 7354312), (48.17758414622325, 4397700), (48.17758414622325, 452513), (47.887746603448946, 1899109), (47.887746603448946, 294036), (46.90178186118888, 2169021)]}, {'query_id': '330975', 'scores': [(75.05599580889046, 7908218), (68.74301967671714, 5750875), (68.61114697605723, 233948), (68.5330334751309, 4108178), (68.15973663074288, 8327008), (68.1111254177718, 3297858), (68.08899292900375, 233945), (67.74974611326772, 3297857), (67.3996258895076, 1097861), (67.33747323942067, 8541461)]}, {'query_id': '332593', 'scores': [(71.8431727999029, 8434873), (70.96516460501167, 5241323), (69.49640767907593, 5241322), (68.0112050334011, 7073995), (66.79701737709765, 4431502), (66.79701737709765, 1650687), (66.16236077687137, 8434869), (66.16236077687137, 7073996), (65.09407417923123, 8434872), (64.27459050314303, 4614346)]}, {'query_id': '336901', 'scores': [(60.34176644669731, 3646550), (59.07907523247901, 2909600), (58.29768973567842, 8528674), (57.95948010645201, 8130980), (57.288980100773244, 7836261), (56.836838589485986, 8528681), (56.836838589485986, 8528680), (56.836838589485986, 8528679), (56.836838589485986, 8528678), (56.836838589485986, 8528677)]}, {'query_id': '3505', 'scores': [(40.22358491932966, 8285655), (40.11525636208674, 8285658), (39.88033014530177, 4520880), (39.84714658497347, 5469617), (39.70094060049665, 8285654), (39.70094060049665, 8285653), (39.6872449489565, 8285656), (39.60667360349221, 502056), (39.599433895422756, 1030851), (39.500524053118625, 1076196)]}, {'query_id': '360721', 'scores': [(54.08096923037398, 6318492), (49.124358954616284, 4284999), (49.00183996322588, 101662), (48.53865442415907, 7043084), (47.770496745609805, 101666), (47.473747434104666, 7826496), (47.17872623267921, 7826493), (47.09467482435841, 1750204), (46.36881422953773, 7826491), (46.24284593278091, 6206175)]}, {'query_id': '384356', 'scores': [(59.99480648980964, 7565188), (46.92340940020695, 2032126), (46.481273719188664, 2032127), (46.020136129606584, 3705729), (45.91836083601846, 5381456), (45.87978014546364, 6863040), (45.861035716616236, 4648730), (45.861035716616236, 412129), (45.83216732113237, 6543899), (45.83216732113237, 2032125)]}, {'query_id': '390360', 'scores': [(53.418343993623616, 604111), (49.684925429118564, 5423359), (49.00225594332635, 2203877), (49.00225594332635, 2203875), (48.35814270952878, 7679050), (47.49045009446167, 6179316), (46.37950467622069, 4683960), (46.37950467622069, 324991), (45.98686260418899, 324994), (45.98686260418899, 324992)]}, {'query_id': '405163', 'scores': [(39.65672194854834, 851536), (38.488027157913606, 6289824), (37.53346868465028, 7622492), (37.2041996029985, 429247), (37.2041996029985, 429246), (37.2041996029985, 429245), (35.08094633910045, 3293026), (35.08094633910045, 2748170), (35.08094633910045, 1518541), (35.08094633910045, 1352548)]}, {'query_id': '42255', 'scores': [(81.99830230475472, 8746197), (76.93941336033014, 8746189), (72.53274575453885, 8746188), (70.56698109788343, 8746194), (67.6566054266488, 3723425), (67.61258658137763, 3063186), (67.26710938112652, 2882385), (67.18728194459419, 8746196), (67.13692256262429, 7123389), (67.06107543765981, 6784918)]}, {'query_id': '425632', 'scores': [(54.9849127050137, 8454642), (54.9849127050137, 8454639), (52.96210694052134, 8454636), (50.669009221373585, 8454635), (50.669009221373585, 650345), (39.10470126086036, 8454641), (33.088593374574145, 8454643), (33.088593374574145, 8454640), (33.088593374574145, 8292459), (33.088593374574145, 6843069)]}, {'query_id': '426175', 'scores': [(36.62081382717152, 330964), (35.82853475260643, 6209130), (35.82853475260643, 4362255), (35.60355811971062, 6758532), (35.60355811971062, 5030738), (34.74455643047207, 6049564), (34.74455643047207, 3017346), (33.89151676948499, 6016624), (33.75483709287114, 3870458), (33.75483709287114, 3562170)]}, {'query_id': '42752', 'scores': [(39.14564253381385, 4303156), (39.03660305752797, 1254236), (39.02441568362146, 7920885), (39.01113857105172, 8660833), (39.01113857105172, 5932129), (38.709879782131125, 5325520), (38.61417208188671, 3818076), (38.60023144801321, 4821761), (38.36203376852673, 4747962), (38.34406982641851, 1986201)]}, {'query_id': '435548', 'scores': [(34.23124563780922, 8547993), (34.23124563780922, 8547992), (34.176488231257274, 7875423), (33.63365758977925, 7875421), (33.110041576373746, 7080768), (33.110041576373746, 235984), (33.063232634841555, 7875428), (33.063232634841555, 6064445), (33.063232634841555, 1394884), (31.996785979958027, 6424374)]}, {'query_id': '436707', 'scores': [(39.40572489481224, 1595287), (38.74514272491306, 2203550), (38.74514272491306, 2101427), (38.74514272491306, 1825031), (38.21323689906292, 5413961), (38.21323689906292, 3370495), (37.41170992684386, 5044105), (36.999759545909825, 8641216), (36.999759545909825, 8200621), (36.79010061606555, 6045618)]}, {'query_id': '444389', 'scores': [(42.94186775902429, 5896071), (42.046252432995615, 8495210), (35.83137367113159, 8044142), (35.83137367113159, 7705876), (35.83137367113159, 5208202), (35.32449516813813, 1722558), (35.05032284274404, 7484162), (35.05032284274404, 7150173), (35.05032284274404, 1048029), (34.60998547165657, 5251228)]}, {'query_id': '449367', 'scores': [(44.8341386768903, 8432230), (44.72002606516108, 635517), (44.14524858103866, 6632652), (43.897347438047056, 6632645), (43.897347438047056, 6632644), (43.141947476924145, 8600273), (42.23984900445052, 5717147), (41.97108294868582, 2596719), (41.908055164774275, 386955), (41.658852702133046, 1998108)]}, {'query_id': '452915', 'scores': [(48.36946137833745, 135840), (46.890192374031805, 2447912), (46.87738946087531, 3332642), (46.531936960579785, 1044828), (46.24523279493054, 953667), (46.041625029853996, 1364570), (45.654370036914926, 8366603), (45.654370036914926, 8366600), (45.654370036914926, 3779965), (45.65074888077799, 7261641)]}, {'query_id': '463271', 'scores': [(53.29601843507277, 6239584), (52.39685443029627, 8604216), (52.39685443029627, 8326637), (52.39685443029627, 6239587), (52.39685443029627, 3914186), (51.53270374689229, 2455683), (51.217919239076224, 8326635), (51.05730677291079, 8435508), (51.05730677291079, 3914192), (50.719716451534886, 6005616)]}, {'query_id': '469589', 'scores': [(62.912578743328915, 8581474), (59.39705823591213, 8581470), (59.39705823591213, 8581468), (55.91797647853005, 2210168), (53.1220227840372, 4871853), (52.514692852944044, 2210175), (49.756696945673234, 8581475), (49.756696945673234, 8581466), (49.756696945673234, 2210172), (49.09149857461225, 6441572)]}, {'query_id': '47210', 'scores': [(72.04149698761682, 2042918), (71.17108161474187, 6180504), (70.45623705082295, 2338385), (68.23727637044001, 5207329), (68.20640307292012, 2545998), (66.95700332559241, 4748997), (65.852782818801, 3858013), (65.55637888637293, 2546002), (65.55637888637293, 2546001), (65.51196960857791, 7367873)]}, {'query_id': '482726', 'scores': [(25.30923542899766, 8730812), (24.957724524421835, 8572751), (24.43052399235153, 6318274), (24.43052399235153, 4734241), (24.43052399235153, 22062), (24.409738663233938, 1272524), (24.262889992818895, 4734239), (24.13162940827919, 4768911), (24.13162940827919, 3512937), (24.13162940827919, 1985718)]}, {'query_id': '48792', 'scores': [(60.142461202325876, 64048), (59.550091466651345, 64047), (54.50735422962718, 64041), (36.69575604269064, 6619302), (36.036718282285875, 5803986), (36.036718282285875, 3820076), (35.72228819898634, 3820082), (35.640934425829776, 6152218), (35.640934425829776, 3820080), (35.6362873153886, 6152219)]}, {'query_id': '50122', 'scores': [(52.51562336202875, 8273511), (49.597684585000025, 1417366), (48.859971433287924, 8273514), (48.413909775096556, 8273517), (48.413909775096556, 425197), (45.408942949956966, 7341000), (39.2229387623488, 8361668), (38.954794166429075, 8337928), (38.954794166429075, 5999140), (38.30410906282784, 8273519)]}, {'query_id': '514096', 'scores': [(26.050940719543505, 4786295), (25.780814184077244, 6286608), (25.780599403454357, 6474925), (25.780599403454357, 6285508), (25.634757394551883, 1837337), (25.56713868266199, 3534826), (25.42108189313663, 8460993), (25.42108189313663, 5936453), (25.400181338647513, 7016672), (25.400181338647513, 3708394)]}, {'query_id': '519025', 'scores': [(33.48802314850731, 4232478), (33.36552953940948, 288148), (32.752431387102305, 2692126), (32.589137505829434, 6805536), (32.589137505829434, 6526749), (32.32687763590634, 6526750), (32.32687763590634, 3394216), (32.32687763590634, 1697202), (32.2954207165846, 6236703), (32.2954207165846, 5931056)]}, {'query_id': '53233', 'scores': [(58.13944861859299, 8333467), (56.41224231051203, 8257054), (56.040393880016076, 8333470), (56.040393880016076, 8333463), (55.34519898821928, 8257048), (55.34519898821928, 5256809), (53.58047349365819, 8257051), (53.58047349365819, 7355044), (53.58047349365819, 5516316), (53.58047349365819, 579087)]}, {'query_id': '537060', 'scores': [(46.57353075342735, 8420027), (46.50644323832398, 8420024), (45.841199619032814, 8420032), (45.841199619032814, 8420025), (43.98220520095438, 8420030), (43.98220520095438, 8420029), (43.98220520095438, 8420026), (43.98220520095438, 7386519), (43.98220520095438, 1086305), (42.41082485029976, 8420031)]}, {'query_id': '537817', 'scores': [(51.42740150537853, 1956211), (51.28097565136481, 6665873), (51.182346891309884, 8527917), (50.997735771926855, 540370), (50.97572565962132, 5264609), (50.832582681357465, 1956207), (50.154353192274726, 3759425), (50.11052674147325, 540368), (50.06688091616198, 2025709), (49.87111331283676, 1095538)]}, {'query_id': '543273', 'scores': [(56.49100254755588, 8507214), (55.259922074494156, 8507211), (52.57248800654146, 8507209), (52.12979659723385, 4308075), (50.90665886078871, 5990522), (49.84291110147114, 8507212), (49.26606709208937, 8507208), (49.26606709208937, 4308073), (49.26606709208937, 3143117), (47.80177076058182, 4067764)]}, {'query_id': '545355', 'scores': [(65.28315715120661, 4290924), (63.222605024751985, 4290920), (61.91094842520934, 8547218), (61.91094842520934, 8547215), (61.60240395507491, 8547219), (61.60240395507491, 4290921), (61.025559945693146, 4290922), (55.4484455142361, 4290923), (48.33499173929434, 4290918), (48.33499173929434, 3294315)]}, {'query_id': '555530', 'scores': [(49.98257931748847, 1897716), (49.105465273627246, 7475084), (48.983391602053935, 4296807), (48.3441786563642, 4137963), (48.20351149968182, 1754575), (48.159110956669025, 6485976), (47.58397897626271, 6150835), (47.56191474971767, 3948935), (47.16302005438847, 4238674), (46.871775018971874, 1483513)]}, {'query_id': '583468', 'scores': [(36.90751119221192, 7491704), (36.76461198511346, 8326078), (36.543671772746784, 7491700), (36.543671772746784, 1675562), (36.41882313250726, 7706119), (36.41882313250726, 2267054), (36.41882313250726, 2267052), (36.41882313250726, 1675564), (36.31500209355011, 6654310), (36.31500209355011, 2080306)]}, {'query_id': '586148', 'scores': [(42.40527172830462, 5056079), (41.09699439133204, 960743), (40.97360954601184, 8271038), (40.89417475961996, 6148926), (40.14103736768357, 7779366), (39.96540013342907, 6854043), (39.96540013342907, 1658943), (39.9382177359715, 2811037), (39.86967718469157, 8567688), (39.805547007565096, 5678812)]}, {'query_id': '590019', 'scores': [(40.20411617629251, 4408763), (40.081109370110084, 2831848), (40.07505796133358, 2831847), (40.00770821803319, 632727), (39.50111262738527, 5075460), (39.23172777241244, 3453506), (39.17817875019908, 3266922), (39.12452349685708, 6066327), (39.11583359074084, 1248322), (39.10995876299576, 8369826)]}, {'query_id': '605127', 'scores': [(54.11548807064422, 1137979), (53.63868742978599, 8642036), (51.67908714869748, 8594907), (51.67908714869748, 1137978), (51.10513675116847, 5121398), (49.3849181350089, 3679424), (48.58270565493352, 8547782), (48.110446601436536, 8642038), (48.110446601436536, 1137982), (46.661053204929736, 6888324)]}, {'query_id': '610265', 'scores': [(57.81068897830704, 474853), (57.23985767433712, 222769), (57.22804056936947, 2877322), (57.22804056936947, 683818), (57.03437888331837, 2877320), (57.01526786569832, 1148168), (56.9202528010967, 4758657), (56.90115476305082, 1148169), (56.85419722015643, 7009719), (56.7922864372504, 2039693)]}, {'query_id': '611953', 'scores': [(76.86529178866935, 484756), (67.09434336461452, 7197690), (61.0807182461033, 3562872), (61.0807182461033, 484758), (60.15195191705084, 456354), (58.04387977170319, 8062950), (58.01152405574676, 456353), (57.50945951221826, 456359), (57.34150813202862, 8062949), (56.58132151609455, 8371935)]}, {'query_id': '640502', 'scores': [(44.100844821861514, 3551380), (43.81058170243895, 5827712), (43.348728971708255, 5827713), (42.467291970351525, 6730887), (42.467291970351525, 3271456), (42.467291970351525, 1861075), (42.44843004963134, 6985038), (42.42397995024684, 5150987), (42.2563746756901, 3678130), (42.14079192459329, 77134)]}, {'query_id': '64647', 'scores': [(49.53074057816176, 8293649), (48.95272005223479, 8293652), (45.97457718718075, 4161653), (43.778861021610396, 8293648), (36.55368203514877, 8599633), (35.33103896278777, 1422031), (35.26874016407929, 6508277), (34.940312891418245, 7509908), (34.940312891418245, 5692182), (34.940312891418245, 1310940)]}, {'query_id': '653399', 'scores': [(42.082893004833, 1969533), (40.02014293866196, 7133996), (39.333402948297895, 8458007), (39.17807835386691, 7485466), (39.17807835386691, 6343750), (39.17807835386691, 3111946), (39.17807835386691, 2007733), (38.5979610792473, 1674561), (38.36022516634597, 2399252), (38.287226667215904, 7092040)]}, {'query_id': '655526', 'scores': [(44.74561000299681, 6040076), (43.480011674325894, 8425101), (42.082366662207335, 206999), (38.463087250365206, 6707899), (38.463087250365206, 3864910), (38.463087250365206, 3021665), (38.463087250365206, 2105608), (34.030331576754556, 1826238), (33.55548974079984, 5883282), (32.79286497396349, 7135528)]}, {'query_id': '655914', 'scores': [(36.017649070385524, 8613490), (34.92954531739092, 1270195), (32.16743579055848, 8613495), (32.16743579055848, 8058173), (32.16743579055848, 5866957), (32.16743579055848, 5235329), (32.16743579055848, 2789189), (26.533999952542953, 3209412), (26.397695158266195, 6248270), (26.397695158266195, 2519335)]}, {'query_id': '660198', 'scores': [(34.829731126439555, 2845619), (33.25390365950289, 523901), (33.23231698228326, 2410610), (32.93869516094842, 4434042), (32.86208509584191, 7327517), (32.86208509584191, 4264632), (32.86208509584191, 3556727), (32.41064196814557, 7327515), (32.41064196814557, 4264630), (32.41064196814557, 3604202)]}, {'query_id': '67316', 'scores': [(69.31739877796132, 6680964), (61.74008977957729, 7630885), (60.67639069017124, 6764671), (59.43372139062002, 6179107), (59.222137775613646, 4163705), (58.84570810567906, 7112302), (58.67993075090326, 4252546), (58.67993075090326, 3810719), (57.706332307261796, 8431637), (57.49030490561766, 1974375)]}, {'query_id': '673670', 'scores': [(31.677507808414084, 3607501), (31.37866339512716, 7257303), (31.37866339512716, 2240359), (30.940821580311432, 8008345), (30.940821580311432, 3607497), (30.940821580311432, 1905068), (30.940821580311432, 1398756), (30.237621089849814, 2240358), (30.237621089849814, 2240357), (30.237621089849814, 2240355)]}, {'query_id': '701453', 'scores': [(44.37313410233498, 4355647), (44.37313410233498, 4330679), (43.7125107955118, 4355649), (43.5450485903323, 8626149), (43.20217902648689, 6999087), (43.001632677822585, 4330677), (42.59095006563356, 2036081), (42.50419993502207, 4355645), (42.39368264533894, 2036076), (42.248080501788145, 4330673)]}, {'query_id': '703782', 'scores': [(40.078870189466954, 4061595), (39.38949832640834, 4061598), (38.59522204940601, 4061597), (38.59522204940601, 344218), (38.5905197888808, 185599), (38.05294653000127, 4647484), (37.95344897359844, 4948640), (37.82236489135344, 3490915), (37.82236489135344, 611402), (37.641815117695224, 7687244)]}, {'query_id': '708979', 'scores': [(37.34653392819264, 6157147), (37.27320518594547, 4805197), (37.18849165746205, 279265), (37.11570850987285, 3811546), (37.08056894309277, 2889574), (36.96273068173775, 6313188), (36.93838504955863, 1447346), (36.89632375002151, 8088373), (36.85480796736847, 6409763), (36.85480796736847, 3422810)]}, {'query_id': '716113', 'scores': [(41.48546173539723, 8520612), (41.48546173539723, 8520611), (40.906594827461454, 8520617), (38.238773425670495, 8520618), (38.238773425670495, 8520614), (38.238773425670495, 8520613), (33.82660726117005, 8520616), (33.82660726117005, 8520610)]}, {'query_id': '730539', 'scores': [(51.34875866378623, 8577877), (49.11445004172529, 8276218), (48.31982754625078, 2666444), (47.85975559080795, 2666436), (46.41412937562057, 8577880), (42.74446282937568, 8577876), (42.74446282937568, 5428552), (42.74446282937568, 4605022), (33.249402949309726, 8276217), (33.249402949309726, 7916064)]}, {'query_id': '735482', 'scores': [(38.41575399332912, 8405801), (37.94984954029286, 8405802), (37.42754360349872, 4823368), (37.10116164799611, 8405807), (37.10116164799611, 338960), (36.70591114879592, 8405806), (35.87446505968181, 8405805), (33.84569762516202, 4117288), (33.763871141161395, 8405804), (33.763871141161395, 8405803)]}, {'query_id': '735922', 'scores': [(38.287478463923726, 5537112), (38.287478463923726, 5537109), (36.87285885862202, 5854095), (36.87285885862202, 281701), (36.87285885862202, 281697), (36.87285885862202, 180247), (35.84158648258619, 8626887), (35.28431209261885, 7371194), (35.28431209261885, 5861444), (35.28431209261885, 1432224)]}, {'query_id': '75198', 'scores': [(44.5680297652721, 7797928), (44.19144063474392, 2579716), (43.46729348042474, 8256960), (41.746862365439476, 6862216), (40.79090534179102, 6862217), (40.79090534179102, 3916717), (40.79090534179102, 2140526), (40.79090534179102, 2140522), (38.58286790175765, 7088445), (38.58286790175765, 2932859)]}, {'query_id': '768208', 'scores': [(37.75961602044883, 52162), (37.2327376573728, 52155), (36.3865390742507, 8525681), (36.3865390742507, 52159), (34.8045156362398, 3515884), (34.8045156362398, 3515882), (34.8045156362398, 813102), (34.8045156362398, 52160), (34.8045156362398, 52158), (34.8045156362398, 52157)]}, {'query_id': '779302', 'scores': [(47.82895060153685, 1583173), (46.79323829560066, 8768560), (46.79323829560066, 8768554), (44.164122442070315, 8768563), (44.164122442070315, 8768562), (44.164122442070315, 8768561), (44.164122442070315, 8768558), (44.164122442070315, 8768555), (36.72228991713043, 2411546), (36.161380922321825, 4638389)]}, {'query_id': '792635', 'scores': [(38.43586318271924, 4294317), (36.76473869651405, 8682263), (36.76473869651405, 8682259), (36.76473869651405, 8682258), (36.76473869651405, 8682254), (36.76473869651405, 6306494), (36.76473869651405, 6306491), (36.76473869651405, 4294318), (36.76473869651405, 4294312), (36.76473869651405, 4294311)]}, {'query_id': '794223', 'scores': [(35.62706082134951, 7790213), (35.13105720992483, 2519053), (33.76895447706966, 8445687), (31.530393861700702, 471681), (31.51624611119379, 6209396), (31.234843268722692, 5946388), (31.234843268722692, 892194), (29.872740535867518, 843980), (29.872740535867518, 843973), (29.872740535867518, 520630)]}, {'query_id': '794429', 'scores': [(46.569008536102196, 1208450), (44.51291509151622, 1208449), (44.20148890909699, 64014), (42.141284933092805, 8618468), (42.141284933092805, 4652685), (42.141284933092805, 719934), (36.31505429909995, 6098420), (36.29910492195128, 1246185), (36.17933196901575, 3926405), (35.91957219536736, 4627633)]}, {'query_id': '801118', 'scores': [(47.140400328519995, 2932991), (46.2526073427832, 4799238), (46.11241858809377, 7615528), (46.11241858809377, 3975963), (44.660632527270224, 1930307), (44.497875184098184, 3263703), (43.84442590055729, 7486725), (43.35650768638846, 4214955), (43.34240916934306, 3360132), (43.3186959572832, 5219693)]}, {'query_id': '804066', 'scores': [(28.777996594823257, 73387), (28.464785772938193, 6152070), (26.953598122714453, 8567803), (26.953598122714453, 7922526), (26.953598122714453, 7621407), (26.953598122714453, 5655169), (26.953598122714453, 1504069), (26.953598122714453, 1259677), (26.286602248327824, 7427197), (26.286602248327824, 3298496)]}, {'query_id': '808400', 'scores': [(40.46393216142154, 3691917), (39.06400961808753, 3691923), (37.94796105262329, 3377142), (36.43127966878727, 7282136), (36.43127966878727, 7252539), (36.43127966878727, 4576228), (36.43127966878727, 1995351), (33.94461212838032, 5834423), (33.7462107843554, 5598346), (33.173143670917135, 5834421)]}, {'query_id': '809525', 'scores': [(50.17209296225508, 3793929), (45.049306736789724, 2077416), (44.03457266566295, 1724922), (43.08702265317709, 1428570), (42.54425791745811, 3793922), (42.54425791745811, 1428575), (41.529523846331344, 7059360), (41.529523846331344, 1428574), (41.529523846331344, 1428573), (41.45870925434113, 2077420)]}, {'query_id': '814183', 'scores': [(56.96149713957998, 8679359), (44.601420126516565, 8444060), (41.33399678576579, 8679357), (40.32149424919948, 7274881), (36.857312996525316, 510865), (36.857312996525316, 510863), (36.019646792058836, 510861), (34.4535751924041, 7013314), (34.4535751924041, 878335), (34.4535751924041, 510868)]}, {'query_id': '819983', 'scores': [(28.598512620719063, 8145644), (28.598512620719063, 1338142), (28.499635414944272, 1617043), (28.43395625277155, 4580581), (28.431089247244948, 8312879), (28.42739108255476, 8328822), (28.375044959941107, 8439123), (28.36344605604693, 8831812), (28.36344605604693, 8045345), (28.361145313121714, 1221184)]}, {'query_id': '849550', 'scores': [(40.688357140999486, 1373164), (40.57660806039334, 7274031), (40.21694598955393, 6758594), (39.554534793403796, 8582102), (39.46437418890504, 7767168), (39.46437418890504, 4061418), (39.06422286267801, 8000555), (39.06422286267801, 7767169), (39.06422286267801, 1419091), (39.055003542353596, 5690724)]}, {'query_id': '85020', 'scores': [(56.88915261791594, 3152840), (56.52826372298955, 198481), (56.02350456410863, 198476), (55.59809281411971, 6290170), (54.99428617341411, 6974021), (54.70556586106912, 1770218), (53.35859334756081, 603541), (53.233173799482316, 147291), (53.031742404083516, 1640375), (52.98375191377768, 1777604)]}, {'query_id': '850358', 'scores': [(53.41830629231691, 2852825), (53.41830629231691, 2852824), (52.309587722334214, 2852830), (51.2972794627848, 6427506), (50.32041532552215, 2852827), (50.26001009482349, 2852826), (45.37836260169424, 8037013), (41.924703243395015, 389081), (41.858754195840305, 2278756), (41.85350544314281, 6427505)]}, {'query_id': '86606', 'scores': [(49.40269421509956, 5013458), (49.246146219700194, 8422274), (49.22244301165062, 6253490), (49.22244301165062, 6253489), (49.22244301165062, 677277), (49.012916106728454, 3077883), (48.83352820458131, 747160), (48.23535297371981, 6897895), (48.02723777980165, 677274), (48.02723777980165, 677272)]}, {'query_id': '877809', 'scores': [(55.869984403361535, 4888926), (55.47780792289071, 5940394), (55.47780792289071, 5940393), (55.06745011143402, 6704838), (54.721671557897515, 741714), (53.25889084621758, 4888929), (52.99297739881336, 7040022), (52.971660347486306, 7040017), (52.9187042974128, 1550610), (52.426733348915484, 7346771)]}, {'query_id': '883915', 'scores': [(58.50242039117779, 4042980), (57.068916864266875, 6172403), (56.36733001062542, 4042985), (56.27862100267711, 3535079), (55.98845923039123, 5406348), (55.93740933056204, 6172404), (55.035817020107885, 4857055), (54.73405602483984, 5559530), (54.659227889579704, 5406347), (54.61587722217311, 1845958)]}, {'query_id': '88495', 'scores': [(27.491116539453373, 1091557), (27.432907154040997, 2875829), (27.381860614867406, 6579243), (27.338157978312843, 6846565), (27.338157978312843, 1667678), (27.307487605962507, 7950301), (27.307487605962507, 5897696), (27.298933517708676, 5915611), (27.213329699178907, 5459872), (27.212738430234346, 3668359)]}, {'query_id': '911232', 'scores': [(89.68853793891248, 8816622), (85.41859984381088, 2999930), (83.84994531565226, 2999933), (81.29627474797786, 8816620), (78.24472828775302, 6928655), (76.2139999783883, 1879470), (72.17482094121382, 2999925), (69.34263652783989, 8816626), (68.59610124927131, 2999926), (67.46043502816724, 2999929)]}, {'query_id': '914916', 'scores': [(44.05395237044219, 6042350), (44.05395237044219, 2631739), (43.66548346270663, 2631743), (43.31529490436992, 7040507), (43.08485645055001, 8520449), (43.08485645055001, 8481901), (43.08485645055001, 5177082), (43.08485645055001, 2001628), (43.08485645055001, 2001626), (42.989938018266656, 6071696)]}, {'query_id': '91576', 'scores': [(39.56531196159263, 8279891), (38.88274507374275, 3780281), (38.74748942566475, 1382984), (37.15007528150846, 4161753), (31.1446280854899, 3540074), (30.77953416873299, 6805821), (30.77953416873299, 6137131), (30.77953416873299, 3255380), (30.096967280883113, 4066736), (30.096967280883113, 3880194)]}, {'query_id': '918162', 'scores': [(69.15507604954739, 2692389), (69.15507604954739, 2692384), (68.99332326678271, 2692388), (67.41761039320298, 267612), (66.21509124188817, 8217238), (64.81162371766727, 4658366), (64.81162371766727, 1290157), (64.81162371766727, 672942), (62.377876758993885, 2334496), (59.17634300056274, 1223048)]}, {'query_id': '938400', 'scores': [(47.320117404810816, 7239123), (45.09719990318791, 6499450), (45.09719990318791, 2969902), (41.5892501823983, 2870510), (41.110066410995756, 3074064), (39.89367683743545, 7029656), (39.89367683743545, 3424110), (39.89367683743545, 2371820), (39.89367683743545, 790219), (39.89367683743545, 47285)]}, {'query_id': '940547', 'scores': [(57.6578866672199, 4481817), (56.54169722074124, 6404591), (55.29245977260089, 4664999), (54.16604689920156, 3822464), (54.14978929217391, 4820847), (53.32637277018608, 7793152), (53.32637277018608, 4100051), (53.32637277018608, 3408094), (53.004557688150726, 1976277), (51.68505170960225, 2947470)]}, {'query_id': '945835', 'scores': [(52.36312382251422, 4431195), (50.867533754185395, 4539379), (50.668356091423504, 5548718), (50.668356091423504, 4431193), (50.24105694857643, 4425412), (50.24105694857643, 61200), (50.24105694857643, 61192), (49.727588705099855, 5548713), (49.727588705099855, 4539378), (49.17849199499683, 61193)]}, {'query_id': '978031', 'scores': [(35.57992070792969, 8744912), (35.048656349165086, 8744911), (34.25643386412537, 4755047), (34.23999172552556, 8744914), (33.222138367648235, 261672), (33.10307759611424, 8802560), (32.94621975425198, 3698979), (32.73017027395113, 5287819), (32.73017027395113, 4543384), (32.73017027395113, 2398979)]}, {'query_id': '985594', 'scores': [(36.03364863340221, 1657406), (35.53085353619195, 8677959), (35.53085353619195, 8677953), (35.53085353619195, 7740126), (35.53085353619195, 4286031), (35.53085353619195, 1657408), (35.53085353619195, 1657405), (35.53085353619195, 655890), (34.723334137642134, 7740128), (34.723334137642134, 7740125)]}, {'query_id': '99005', 'scores': [(82.82740211642795, 8257322), (81.26110407227809, 6493343), (79.35818902203503, 6493344), (79.03984766449662, 2875400), (79.03984766449662, 1375045), (76.64975190985699, 297028), (76.64975190985699, 297023), (71.74403287666426, 6327640), (71.56534267020356, 8257320), (70.84899092185182, 8476970)]}, {'query_id': '997622', 'scores': [(50.81806457480276, 4543865), (49.597337128496235, 73465), (47.44904533229087, 4518222), (47.44904533229087, 2275574), (47.03632455396879, 73459), (46.49856745710275, 4558331), (45.891441514696886, 4643397), (44.35027566089739, 8533344), (44.35027566089739, 3913148), (43.73199729370113, 5430569)]}, {'query_id': '999466', 'scores': [(40.5782020099497, 8748919), (40.5782020099497, 8748914), (37.93179753103994, 8748922), (37.93179753103994, 8748918), (33.55505166207379, 8748921), (33.55505166207379, 8748920), (33.55505166207379, 8748916), (33.55505166207379, 8748915), (33.55505166207379, 8748913)]}, {'query_id': '132622', 'scores': [(42.18282807056723, 5901873), (41.62670785372853, 962815), (40.40834526053401, 2459821), (40.40834526053401, 821354), (40.17642607512765, 2979805), (39.09006148784269, 7122663), (38.51185848962395, 5656719), (38.37160767508386, 3162996), (37.829143681034324, 6358747), (37.31557867780947, 3358937)]}]\n"
          ]
        }
      ],
      "source": [
        "bm25_results = query_processing(trec_dl_2020.queries_iter(), taat_bm25)\n",
        "print(bm25_results)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VgEMmL6NrwjB",
        "outputId": "a103b69c-72a7-4995-abf8-e07c05a25238"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "query_processing (306289.325 ms)\n",
            "[{'query_id': '1030303', 'scores': [(37.13602502160565, 8726436), (32.82578122640052, 8726435), (32.82578122640052, 8726433), (32.82578122640052, 8726429), (30.70863607555851, 8726437), (30.70863607555851, 8726434), (30.70863607555851, 8726430), (30.70863607555851, 7156982), (20.09607619370668, 1305521), (20.09607619370668, 1305520)]}, {'query_id': '1037496', 'scores': [(27.164761525836614, 7766587), (27.164761525836614, 7766585), (27.164761525836614, 5927420), (27.164761525836614, 4760912), (25.1453728825241, 4905511), (25.1453728825241, 4760914), (25.1453728825241, 4174377), (25.1453728825241, 3725937), (18.72239393334702, 4174376), (18.296884980316406, 4174378)]}, {'query_id': '1043135', 'scores': [(36.87987445266299, 8696961), (35.933761717917605, 650642), (34.965363265314046, 3378240), (34.82276430970873, 8696958), (34.312225997811325, 4994428), (33.41670021144713, 4355523), (33.41670021144713, 2514458), (32.78034666132486, 650641), (31.436423214371146, 2664995), (30.82455348018611, 4721484)]}, {'query_id': '1045109', 'scores': [(34.19487039697934, 8838896), (30.129545984624254, 8838890), (28.860477596135542, 8838893), (28.701724152162242, 8838892), (28.153437726491497, 8838891), (26.761633722865753, 8838899), (26.761633722865753, 8838898), (26.761633722865753, 8838897), (26.761633722865753, 8838895), (21.013377074207764, 7620329)]}, {'query_id': '1049519', 'scores': [(28.56064263010628, 7105), (27.818609848717962, 5355123), (25.53010512617196, 1316185), (25.362403072758422, 7106), (24.777212305223607, 7668125), (22.65232350353714, 5355121), (22.488707582677602, 6080565), (22.0739416296838, 5355126), (22.00765545704307, 6950445), (22.00765545704307, 6580258)]}, {'query_id': '1051399', 'scores': [(29.575974242084552, 4426188), (29.575974242084552, 69818), (29.575974242084552, 69813), (29.452244366377762, 2845499), (29.019131142230236, 7848674), (28.98430649216766, 3773237), (28.68125490244961, 8712317), (28.485325955804058, 2866972), (28.25058202040101, 3446937), (28.12172342732147, 3943228)]}, {'query_id': '1056416', 'scores': [(43.48907029835839, 8739209), (41.02607316236592, 2308626), (37.62130424677382, 8757245), (37.62130424677382, 2059786), (35.565564644482905, 3790242), (35.44467447809854, 895432), (34.417165894385676, 7321435), (34.29627572800131, 2348501), (34.29627572800131, 1367298), (32.26903594513811, 2348504)]}, {'query_id': '1064670', 'scores': [(32.914793712531775, 8680901), (32.06413070969825, 8680903), (31.69183135257443, 8680905), (31.077692190131927, 8680897), (29.796618308113246, 8680904), (28.573655948155896, 8680902), (24.724207855807858, 7734392), (23.50124549585051, 3807086), (22.887222815463574, 129462), (22.887106333408013, 8337453)]}, {'query_id': '1065636', 'scores': [(32.13692641557028, 4371749), (31.25511911696255, 1062134), (31.175178692601186, 8490752), (26.858925790916082, 7411479), (26.39675824713418, 436583), (26.245829481570585, 3822992), (26.245829481570585, 597867), (26.102079769511043, 6307138), (25.83717725185279, 7411495), (25.762702017570106, 444919)]}, {'query_id': '1071750', 'scores': [(52.60972937985006, 3928913), (50.998790298666265, 8698268), (48.8128472491279, 8698276), (47.5994270983294, 8698274), (47.5994270983294, 8698269), (47.5994270983294, 3575934), (46.1699817709662, 8378390), (40.48248232507933, 8378386), (39.05303699771613, 8378389), (39.05303699771613, 8378385)]}, {'query_id': '1103153', 'scores': [(25.614751592222284, 8226438), (25.322313885119613, 8226436), (24.930172896167527, 8226445), (24.902565344478262, 7570096), (24.54923606597886, 6113506), (23.837049818234835, 8226444), (23.837049818234835, 8226440), (23.837049818234835, 8226439), (23.09470794635881, 8226443), (23.09470794635881, 8226441)]}, {'query_id': '1103791', 'scores': [(17.30903162087153, 8384424), (17.038590400484036, 8384416), (16.879445471964058, 4034845), (16.780441962841426, 8220282), (16.694099819008517, 4034852), (16.55600112945586, 1512326), (16.5439328936394, 8384417), (16.5439328936394, 8146212), (16.383198960767587, 8220278), (16.383198960767587, 8214165)]}, {'query_id': '1104501', 'scores': [(36.07442617891704, 6871582), (35.81297390744045, 6067096), (35.802183908778545, 5286080), (35.373549001613824, 2204057), (35.338146425164496, 2204054), (35.27536209957584, 2090898), (35.210441753584206, 5594847), (35.14479513430444, 7431269), (35.121782516435864, 2283306), (35.09948763109822, 3604439)]}, {'query_id': '1105792', 'scores': [(21.51770058291308, 8201616), (21.51770058291308, 8201611), (21.31470340760258, 8201618), (21.31470340760258, 8201612), (21.017288941449983, 8201613), (20.539623283689757, 8201615), (20.539623283689757, 8201610), (20.539623283689757, 6977017), (19.6465961843989, 8201614), (19.6465961843989, 7102720)]}, {'query_id': '1105860', 'scores': [(28.313045350974456, 3593906), (28.313045350974456, 3593905), (28.313045350974456, 3593904), (28.066687266498956, 8200940), (28.026995388461547, 8282567), (28.026995388461547, 8282566), (28.026995388461547, 8282565), (27.824003562189137, 7237676), (27.80393170478729, 3593907), (27.686325329248515, 4866505)]}, {'query_id': '1106928', 'scores': [(32.35964120893993, 4285175), (31.49798753258775, 2168463), (30.277382400824493, 5186636), (29.934481189841122, 7744803), (29.62139747546501, 8190627), (29.62139747546501, 8190624), (29.118554103751585, 2866124), (29.05421131544879, 8190620), (28.87455403454204, 1392371), (28.155189259162842, 6949062)]}, {'query_id': '1106979', 'scores': [(33.04043816379437, 8190140), (33.04043816379437, 8190137), (32.94120133298984, 7674419), (30.375706989944092, 7674423), (29.694403774851565, 5383176), (29.694403774851565, 4219185), (29.694403774851565, 1589841), (28.76853017485402, 8190135), (27.21073850578742, 8190134), (26.829894021844776, 348057)]}, {'query_id': '1107315', 'scores': [(21.61450985364886, 4598318), (20.507081108128187, 8187302), (19.769500198521722, 8817722), (19.769500198521722, 2696421), (19.769500198521722, 2696417), (19.769500198521722, 2696416), (19.319920793207245, 2687026), (19.319920793207245, 2687024), (19.319920793207245, 1708353), (18.675564336690158, 6437724)]}, {'query_id': '1107440', 'scores': [(33.799493738424545, 8186231), (31.856703544081675, 8186223), (25.06835933046453, 789989), (23.958479599927493, 1409567), (21.142673285174283, 3971063), (20.58385212750857, 1879424), (20.408754161530688, 8186227), (20.408754161530688, 5177729), (20.408754161530688, 5177724), (19.678793695377344, 2888123)]}, {'query_id': '1108450', 'scores': [(30.97590845311654, 1629571), (28.625692970081587, 4400204), (17.62204540010716, 1629577), (17.33705275212699, 6285212), (17.33705275212699, 6285210), (17.09514038814382, 8177145), (17.09514038814382, 4747082), (17.09514038814382, 4400207), (17.09514038814382, 4400205), (17.09514038814382, 1629573)]}, {'query_id': '1108466', 'scores': [(32.370544026766346, 8177012), (30.390105521457272, 4944265), (30.390105521457272, 3267306), (30.390105521457272, 753377), (30.13155264408124, 2702004), (29.924987560214298, 841054), (29.758691159685338, 5380573), (29.758691159685338, 3482588), (29.758691159685338, 3267303), (29.69511183471414, 2002688)]}, {'query_id': '1108473', 'scores': [(39.34059712381017, 8176948), (37.68282511102558, 1188068), (37.16912082641594, 1188069), (35.07552720572149, 8176949), (31.163909852171926, 1970473), (30.894958052764473, 3978700), (30.280919883963882, 8176947), (30.006105571426232, 1188063), (29.324416850652305, 5562098), (29.32287310250773, 357762)]}, {'query_id': '1108651', 'scores': [(28.592520796722653, 6348741), (28.40126913590069, 8175412), (26.83656060387166, 1371583), (25.879959013593783, 5027435), (24.05154746508245, 6985088), (24.05154746508245, 5997928), (23.73405119966072, 6663873), (23.64573000746448, 1325612), (23.55031615206348, 6985085), (23.55031615206348, 4617850)]}, {'query_id': '1108729', 'scores': [(34.46548470515693, 3405694), (31.260368667882965, 7789132), (29.328741387876683, 5449883), (26.673919335707872, 3405688), (25.126006579146264, 8174842), (24.446233335738857, 8322380), (24.29624851460401, 334958), (24.13167319151686, 7993178), (23.92890109383491, 1820798), (23.871601357354894, 5623921)]}, {'query_id': '1109699', 'scores': [(17.838621470487702, 2769074), (17.82912688788502, 4872045), (17.780860005807057, 8394502), (17.717270304021795, 7722707), (17.717270304021795, 7722706), (17.717270304021795, 7471054), (17.717270304021795, 2341119), (17.717270304021795, 737337), (17.63771206021638, 7744060), (17.63771206021638, 178437)]}, {'query_id': '1109707', 'scores': [(34.94151316497033, 1911564), (33.949340895762504, 1719374), (33.840084503169706, 5947139), (33.83131666002568, 4122615), (33.36118900149726, 771002), (33.25338687566345, 4122618), (33.241440091836026, 1719371), (33.05516998358234, 5740304), (32.8663855939665, 4285843), (32.7158712336693, 4099609)]}, {'query_id': '1109850', 'scores': [(33.016789385490725, 8164028), (31.88684380605354, 8164035), (31.88684380605354, 8164030), (31.044837943352107, 8164033), (25.71940524207831, 3633621), (25.71940524207831, 3633617), (23.38605687821022, 8164032), (22.22063876135589, 8164036), (18.3747589757366, 8164029), (15.321955960837355, 5382247)]}, {'query_id': '1110678', 'scores': [(26.0675417177549, 7139329), (25.007542116324885, 8155914), (25.007542116324885, 6632692), (25.007542116324885, 2342590), (25.007542116324885, 2342589), (24.534580786659845, 8155909), (24.534580786659845, 5719597), (23.254465852421404, 2714688), (23.254465852421404, 2126363), (23.254465852421404, 2096160)]}, {'query_id': '1112142', 'scores': [(44.306069518791695, 2542782), (43.8110027055872, 3651292), (43.8110027055872, 3651291), (43.8110027055872, 3651290), (43.58272649544322, 7866090), (39.649360688789024, 2238202), (37.931331524148405, 1946438), (37.23978350620301, 5670185), (37.23978350620301, 4967524), (37.23978350620301, 3974360)]}, {'query_id': '1113042', 'scores': [(27.768761791257056, 7072957), (27.537084150153333, 2410652), (26.93764472193051, 355186), (26.429509473953956, 2499270), (26.17508316216948, 3521102), (25.699416579268068, 7834150), (25.699416579268068, 4931496), (25.699416579268068, 4243566), (25.40616954106973, 2326525), (25.40616954106973, 2326519)]}, {'query_id': '1113256', 'scores': [(50.37766796768834, 7561115), (50.02546094432704, 8130535), (50.02546094432704, 7561120), (48.7793839656685, 7561117), (47.69589360057542, 7561116), (47.34017985027023, 7561114), (46.96111350052801, 8130530), (46.96111350052801, 7561113), (45.52541611207362, 7561111), (41.880953510612486, 8130528)]}, {'query_id': '1114166', 'scores': [(23.973880881167275, 6019023), (23.733030202193945, 3594677), (23.733030202193945, 3142379), (23.733030202193945, 2463783), (23.105358041182587, 8478109), (22.853308585460223, 5989751), (22.707228582310428, 5193359), (22.70631790383011, 1816877), (22.430263738038953, 5005484), (22.430263738038953, 3951381)]}, {'query_id': '1114286', 'scores': [(14.631235349876171, 6846966), (14.631235349876171, 4956824), (14.539598623409951, 2589532), (14.484065179475694, 4956825), (14.239105916662165, 8720406), (14.239105916662165, 3960437), (14.239105916662165, 2177460), (14.196073311981468, 8413644), (14.196073311981468, 6040011), (14.151533137296873, 6269131)]}, {'query_id': '1114993', 'scores': [(17.420279841642113, 4104358), (16.931023600474244, 8113957), (16.65461327441823, 6612208), (16.65461327441823, 6203023), (16.65461327441823, 6011208), (16.65461327441823, 4956693), (16.65461327441823, 3271410), (16.65461327441823, 2086636), (16.65461327441823, 1881796), (16.65461327441823, 1724571)]}, {'query_id': '1115210', 'scores': [(30.682138596822853, 8111887), (28.280927750114977, 8111878), (25.01774377894786, 8111885), (25.01774377894786, 8111884), (25.01774377894786, 8111882), (18.0045833542173, 8390254), (18.0045833542173, 7440361), (17.585872113421548, 7440364), (17.585872113421548, 7440363), (17.186193201752875, 7440366)]}, {'query_id': '1116380', 'scores': [(28.14200163383135, 8100548), (28.07687041053985, 8100546), (28.07687041053985, 8100543), (22.27332852011519, 521341), (19.70332907548651, 4118339), (19.70332907548651, 1422757), (19.70332907548651, 137869), (16.487795461508387, 5104772), (16.487795461508387, 1645711), (16.487795461508387, 444969)]}, {'query_id': '1117817', 'scores': [(24.499666748024932, 5310910), (24.499666748024932, 3516296), (23.052243389077656, 4486487), (22.002804946450745, 7531176), (19.95075308603358, 6537918), (19.809901772641894, 1005145), (19.38055600523617, 1035058), (19.225187363697685, 109686), (18.934715555604008, 5970406), (18.934715555604008, 5447234)]}, {'query_id': '1117886', 'scores': [(24.729390226048963, 2921720), (24.010935046127265, 556178), (23.834792168767095, 8456607), (22.876998560548586, 7460949), (22.876998560548586, 7460948), (22.876998560548586, 7024723), (22.876998560548586, 5376945), (22.787221349636653, 5094343), (22.6884832105242, 5301093), (22.204377160801492, 8407176)]}, {'query_id': '1118370', 'scores': [(15.470710112566096, 8656398), (15.253613147182286, 7476349), (15.253613147182286, 2951456), (15.107415753106865, 2224024), (15.089710843157427, 8081645), (15.089710843157427, 2224022), (14.872613877773615, 6211291), (14.872613877773615, 3608410), (14.872613877773615, 1284328), (14.650733424056131, 861060)]}, {'query_id': '1118426', 'scores': [(23.200704994973073, 8081194), (23.200704994973073, 8081193), (19.862302498626043, 8081199), (19.862302498626043, 8081196), (19.862302498626043, 2424347), (18.975664053272745, 6622100), (18.975664053272745, 5598458), (18.975664053272745, 2424353), (17.962655829523936, 6208983), (17.654318509251635, 2469356)]}, {'query_id': '1119118', 'scores': [(25.880494569290445, 8074735), (25.400849993589386, 8074733), (21.676724349900397, 6026130), (21.676724349900397, 5472178), (21.676724349900397, 3609036), (20.624716755145535, 8074741), (20.624716755145535, 8074734), (17.45168340820007, 5451461), (17.45168340820007, 5185340), (17.45168340820007, 4246892)]}, {'query_id': '1119543', 'scores': [(39.938129810117054, 8070706), (36.71542496427709, 8070713), (34.77740545218565, 1603603), (34.66043271334337, 405831), (34.21545751304812, 405832), (33.792449464323, 5052354), (33.67638909629049, 8070707), (31.893197993966883, 4770699), (31.546146272654095, 3612262), (27.51764225045892, 3612261)]}, {'query_id': '1120588', 'scores': [(27.748740126153486, 8060866), (27.350268133712483, 8060864), (26.83288152358641, 8060867), (26.23782631140057, 8060861), (25.66681007500109, 8060862), (25.51687160982233, 8060868), (24.156285305606275, 8060859), (23.78140105033075, 8060863), (23.78140105033075, 8060860), (23.21038481393127, 2313348)]}, {'query_id': '1121353', 'scores': [(31.50922075538937, 8053113), (30.788977921776272, 8053117), (26.584136568730628, 1854970), (24.74400697036654, 4872676), (24.153152654882824, 4210422), (24.153152654882824, 623601), (22.942093217390756, 6379159), (22.538424503581204, 5606064), (22.08151265489449, 5277905), (22.08151265489449, 937044)]}, {'query_id': '1121879', 'scores': [(34.53448664990921, 2603303), (34.527763517602764, 3024534), (33.80231069527568, 1307737), (33.62102190835542, 8048097), (33.62102190835542, 2603307), (33.62102190835542, 1406295), (31.217496305546106, 2193862), (31.154278788903063, 4995632), (31.10461838161462, 8048100), (30.385677869191586, 1025282)]}, {'query_id': '1122138', 'scores': [(15.26907931179721, 4342247), (15.224717169499295, 4342246), (15.197882472090502, 5871298), (14.661510841195334, 5463811), (14.50358063031867, 6283517), (14.359802270608938, 7067825), (14.359802270608938, 6963235), (14.090999250282046, 3967084), (14.090999250282046, 3647404), (14.090999250282046, 3345300)]}, {'query_id': '1122767', 'scores': [(30.243157098170446, 7544609), (30.243157098170446, 1575377), (29.51085732239147, 8039590), (29.51085732239147, 5877496), (28.735454187473536, 5877500), (27.651942506952526, 6851717), (27.651942506952526, 6179207), (27.651942506952526, 2859907), (27.280171233861253, 2019371), (26.23332940743189, 4550504)]}, {'query_id': '1122843', 'scores': [(42.28235712772559, 2332417), (42.26654329362261, 2332424), (41.473422277210375, 8772747), (41.24145930847786, 225969), (40.186842428067926, 7998359), (39.77492604532857, 4947308), (39.57683649216069, 2332416), (39.311821503388394, 3880592), (39.26386769953681, 2332425), (39.153128591318264, 3312892)]}, {'query_id': '1123657', 'scores': [(34.4669201973063, 8032346), (27.298421137384018, 8032342), (25.899726568369065, 8032349), (24.78104788750842, 7221025), (24.466361526784414, 5571459), (24.148603313839708, 8032348), (24.148603313839708, 8032347), (24.148603313839708, 8032343), (24.14075555668393, 7988749), (24.11868999377082, 5572500)]}, {'query_id': '1124552', 'scores': [(30.29621004688822, 5782491), (25.335584446195583, 3442729), (24.081040215422888, 4743535), (24.081040215422888, 3442725), (23.369063525289075, 7082651), (23.3022342623024, 8214271), (23.25592894279268, 2601066), (23.09182536281873, 2601070), (23.09182536281873, 1797406), (22.78502301765006, 6263843)]}, {'query_id': '1125632', 'scores': [(34.75759741403528, 7011199), (34.673996762661176, 6721396), (34.538599585805486, 1278069), (34.3998819242041, 7191566), (34.3998819242041, 1475286), (34.3998819242041, 1243662), (34.32533635566454, 6934400), (34.32533635566454, 6543904), (33.972843627735315, 7011204), (33.972843627735315, 5847127)]}, {'query_id': '1125755', 'scores': [(17.586188788934777, 1744931), (16.930797349218324, 7420642), (16.930797349218324, 2206347), (16.550873167654295, 7856574), (16.550873167654295, 3808440), (16.389609638094438, 7254544), (15.895481727937844, 7387332), (15.895481727937844, 7254547), (15.895481727937844, 2938068), (15.895481727937844, 482655)]}, {'query_id': '1126523', 'scores': [(23.72655829268987, 8805840), (23.07116685297342, 8805844), (23.07116685297342, 8805841), (20.988878489687192, 8805837), (18.04649914848064, 8805846), (18.04649914848064, 8556233), (18.04649914848064, 3214173), (18.04649914848064, 3025043), (18.04649914848064, 2841635), (15.96421078519441, 2774244)]}, {'query_id': '1126738', 'scores': [(40.10230637835696, 8790091), (40.10230637835696, 1091656), (39.295966924466946, 1091662), (39.295966924466946, 1091658), (36.08777408497411, 8790089), (36.08777408497411, 1091654), (35.32548892931486, 8790093), (35.32548892931486, 8790090), (35.32548892931486, 8790088), (34.20150059965, 8790087)]}, {'query_id': '1127004', 'scores': [(27.731932281356613, 6481277), (27.586924376055833, 6531608), (27.498631546733275, 695631), (27.497402001142227, 5224906), (27.353623641432492, 1756214), (27.297674131576997, 3961675), (27.229828526406383, 6011954), (27.153895771867266, 860288), (27.09457195340667, 6654716), (27.09457195340667, 4964550)]}, {'query_id': '1127233', 'scores': [(18.296917757692604, 8100081), (18.12342886544065, 8100077), (18.028882008816563, 8527129), (17.952526984327527, 5400207), (17.897534265338283, 6611628), (17.834510131956637, 124809), (17.785161046384218, 6438836), (17.785161046384218, 3842712), (17.785161046384218, 124808), (17.712858897754856, 6438837)]}, {'query_id': '1127540', 'scores': [(20.029711307171908, 8726963), (20.029711307171908, 8726962), (20.029711307171908, 186476), (19.11262462801214, 8726961), (19.11262462801214, 8726956), (19.11262462801214, 8726954), (17.866149108793955, 2574460), (17.866149108793955, 2044372), (17.866149108793955, 2044369), (17.866149108793955, 2044368)]}, {'query_id': '1128456', 'scores': [(36.6756963878081, 8649622), (31.52360483130378, 1459777), (27.698740155484423, 7526299), (26.940393668245, 4084616), (26.353199966557405, 8649615), (26.353199966557405, 3721820), (25.015360277560312, 4071160), (25.015360277560312, 3121928), (25.015360277560312, 3121927), (25.015360277560312, 2534794)]}, {'query_id': '1129081', 'scores': [(20.33838457106435, 8599986), (17.991647889787693, 7871513), (17.991647889787693, 6962228), (17.991647889787693, 6794362), (17.991647889787693, 4178203), (17.991647889787693, 3762080), (17.991647889787693, 2770798), (17.991647889787693, 2156392), (17.991647889787693, 1044198), (17.991647889787693, 687772)]}, {'query_id': '1130705', 'scores': [(16.243999726377528, 1468216), (16.16435490449481, 8442374), (16.16435490449481, 1621163), (16.108167119193993, 6762105), (16.108167119193993, 6762102), (16.03414194173418, 6628960), (16.03414194173418, 6233460), (16.03414194173418, 5379660), (16.03414194173418, 3570934), (15.93218273504425, 1910013)]}, {'query_id': '1130734', 'scores': [(34.952018860958475, 2546803), (34.08446194588046, 7212169), (33.509625508736946, 8460898), (33.446712141583916, 8460901), (33.25329924293893, 8460902), (30.666221222600228, 4311761), (30.666221222600228, 2810717), (30.666221222600228, 1081476), (30.408492042518155, 3823605), (29.862033728075932, 7427232)]}, {'query_id': '1130847', 'scores': [(36.87746220608539, 5474954), (35.13408642648099, 4201753), (34.131176649782695, 7332266), (34.131176649782695, 7332262), (33.684022437259614, 7332261), (33.59323396645793, 6209899), (33.43552579006536, 8299223), (33.08618191411966, 8299227), (33.03777405474772, 1458056), (32.40360029462333, 6149589)]}, {'query_id': '1131069', 'scores': [(30.09632102857106, 8432048), (29.600353982758953, 8432050), (28.455564586708732, 8191743), (28.455564586708732, 1780971), (28.455564586708732, 1780963), (24.928261880747293, 6907802), (24.62511391822229, 1780966), (24.157753746761543, 6809701), (24.157753746761543, 6809698), (23.953644009481273, 6809699)]}, {'query_id': '1132044', 'scores': [(28.64358307035746, 5858455), (28.13434697724154, 1014774), (27.762729668787674, 5858453), (27.71484125639293, 2121532), (26.354379477910225, 7645045), (26.19608102011078, 1691946), (25.328223732643163, 1818480), (25.236431753714413, 6166188), (25.15032003085051, 1650397), (25.10216726717845, 5810405)]}, {'query_id': '1132247', 'scores': [(39.440453116224546, 6246993), (37.55756787383823, 1210363), (37.55756787383823, 1210358), (37.55756787383823, 1210357), (36.429401632043366, 7390343), (35.698240637802556, 7390342), (35.698240637802556, 6525827), (35.697923428776654, 8238483), (35.26892143881293, 4623273), (34.90816340044835, 7390347)]}, {'query_id': '1132532', 'scores': [(32.61081477279093, 23966), (30.98069213940423, 6994907), (28.976786804282227, 4893092), (28.458069774306416, 6832793), (28.4390716070335, 8374347), (28.349024427076607, 3299150), (28.349024427076607, 2762578), (28.26730788979092, 5989422), (27.807227572517846, 1976995), (27.43803491711387, 6832795)]}, {'query_id': '1132842', 'scores': [(33.24534220672271, 3158781), (31.905914808862367, 7364932), (31.66055379503093, 834533), (31.31756706426125, 6536692), (31.263533374381886, 6570513), (31.06946968394593, 4874198), (31.006281745859162, 4132635), (30.88689219887928, 2324752), (30.77069406288725, 2941398), (29.304463280247305, 2391268)]}, {'query_id': '1132943', 'scores': [(26.466853964310864, 423711), (26.15561141626224, 5552077), (26.13011167012531, 6253953), (24.896488055394506, 622662), (24.736196475763812, 3437105), (22.212929025578777, 6469809), (21.979432678513913, 5782177), (21.908632296799873, 2107961), (21.904153352309205, 2248898), (21.904153352309205, 2107962)]}, {'query_id': '1132950', 'scores': [(22.216822423622848, 8274826), (21.584983949793042, 8619165), (21.292815493813052, 5867940), (20.67975815261899, 8274822), (20.67975815261899, 2303442), (20.45687669624945, 5845390), (20.44689964934932, 2132514), (20.44689964934932, 261777), (20.03470800062943, 6590922), (19.86792810588271, 6625752)]}, {'query_id': '1133485', 'scores': [(21.195613774906562, 5221748), (21.114944497121986, 8330714), (21.09727950941506, 3156351), (21.07712397359886, 3183897), (21.069207185748333, 2418248), (21.05401115419729, 6195711), (21.029694928400236, 8692792), (21.011153945298688, 5630061), (21.004189028734253, 4090344), (20.90125340724906, 3089617)]}, {'query_id': '1133579', 'scores': [(25.26114552648168, 8220878), (24.979719727590098, 4931644), (24.303968783094316, 1817055), (23.777824898210056, 4931642), (23.777824898210056, 4931641), (22.442228882734632, 670206), (21.84940301294373, 8220879), (21.84940301294373, 5335041), (21.84940301294373, 3691486), (21.84940301294373, 2448461)]}, {'query_id': '1134094', 'scores': [(23.896471655686263, 8172161), (22.5243037759774, 8172166), (22.5243037759774, 7182375), (21.789909141496715, 6426847), (21.789909141496715, 6426846), (21.789909141496715, 3445457), (21.104724157728796, 8172159), (19.92567660781497, 7182378), (19.92567660781497, 6426844), (19.92567660781497, 5357491)]}, {'query_id': '1134207', 'scores': [(15.228319401635972, 1698802), (15.191327174863668, 374550), (14.970134895081006, 7155676), (14.75899680983392, 5592724), (14.650529325877994, 8817805), (14.650529325877994, 853685), (14.622444896144334, 3225711), (14.601081406590058, 7049025), (14.4462067914618, 8246744), (14.364260389589369, 3362951)]}, {'query_id': '1134431', 'scores': [(32.93922278649441, 3128065), (32.20525197313406, 8141222), (31.6648153828383, 8141223), (30.926817660769352, 8141221), (30.889761566462973, 7974008), (29.794740508641226, 8141220), (27.57571512507313, 5010274), (27.498461198349204, 2339181), (27.43205066860595, 6720464), (27.43205066860595, 3243219)]}, {'query_id': '1134680', 'scores': [(44.4432504211804, 8118937), (38.3827845939378, 8118936), (37.10187389812646, 8118930), (31.602573354150778, 8796589), (31.402468424526234, 8307236), (31.186372891495466, 8063270), (31.186372891495466, 3526958), (31.186372891495466, 2456858), (31.186372891495466, 2200409), (31.186372891495466, 1865327)]}, {'query_id': '1134939', 'scores': [(23.4130160178025, 322652), (23.39974024612587, 8093333), (22.62931932865123, 8093330), (20.018244021499164, 6204714), (20.018244021499164, 2090588), (18.131766708937956, 8455233), (18.131766708937956, 8455232), (18.131766708937956, 7543137), (17.71968110191664, 8093327), (17.71968110191664, 7543128)]}, {'query_id': '1134988', 'scores': [(43.88708698233529, 8088407), (42.73195457235513, 8088409), (40.66586177165798, 380755), (40.38742672882627, 8088404), (38.30234311034466, 380753), (38.30234311034466, 380752), (38.208489260602846, 7296294), (38.08360338046426, 380748), (37.72856232384763, 6432242), (37.1472107003645, 380749)]}, {'query_id': '1135268', 'scores': [(23.76332407435382, 8386270), (23.609666174644108, 2966472), (23.459880297612425, 5717144), (23.443873795950072, 6259848), (23.443873795950072, 6087160), (23.182308760782814, 4260), (23.15220180262491, 5225950), (23.15220180262491, 1683963), (23.0652654313396, 8371436), (22.968126755194888, 3622462)]}, {'query_id': '1135283', 'scores': [(19.160024054821314, 8061944), (17.86005783676075, 2582837), (17.464027571394123, 4502908), (17.227330631952874, 3368379), (17.12953803272207, 7770955), (17.12953803272207, 7043720), (17.12953803272207, 6979076), (17.12953803272207, 5959072), (17.12953803272207, 5834915), (17.12953803272207, 3989820)]}, {'query_id': '1135413', 'scores': [(37.89640488425962, 8050662), (36.352661311940224, 8050667), (32.74600576276093, 8050665), (32.57778997973306, 2115732), (31.771218646641586, 6672865), (31.603703236152995, 7522683), (31.402804761812433, 6039815), (31.402804761812433, 5287000), (31.402804761812433, 2061136), (31.349346448257897, 5118096)]}, {'query_id': '1135626', 'scores': [(28.431714333972817, 2003615), (28.25362000073719, 4035834), (28.234683243882827, 7676917), (28.2269101200263, 6532689), (28.075073105452102, 2902886), (27.53878171581926, 4619335), (27.036688180794464, 4892864), (26.684926846370335, 7714006), (26.642227741438248, 4079092), (26.516447282662917, 4035841)]}, {'query_id': '1136043', 'scores': [(26.52638863798932, 8728069), (26.487498712623086, 8728071), (25.44249113747719, 8728070), (25.434675022968655, 8728072), (24.902983900265028, 1056540), (24.208461581961334, 8728076), (24.150919448554262, 1056534), (23.77760495160242, 7018326), (23.77760495160242, 5326197), (23.46272150882153, 8728077)]}, {'query_id': '1136047', 'scores': [(32.326042468983964, 8724031), (31.027048056717845, 8724032), (30.26568087593763, 3988779), (29.782331669746572, 3597758), (29.782331669746572, 3006066), (29.126951142344993, 7009393), (28.50255000742722, 8328670), (28.361423901872143, 512482), (28.17470823960782, 2815435), (27.837113234677368, 8724036)]}, {'query_id': '1136769', 'scores': [(41.12814666903049, 8082043), (40.44931038635414, 189017), (39.45554432712372, 129247), (38.89953932790267, 189019), (38.27233292298898, 8082050), (38.267863572752134, 4713242), (37.74423966245326, 189016), (36.66437896352382, 8082046), (36.66437896352382, 5718075), (34.81155589631244, 5508127)]}, {'query_id': '1136962', 'scores': [(61.70544407762314, 7631079), (61.6448369503907, 6185710), (61.18298821151008, 6185711), (60.850036127471114, 6185717), (60.48410976984094, 8336574), (59.97488752073806, 6185714), (59.97488752073806, 6185712), (56.60620692951767, 596001), (54.77170492407001, 6185716), (54.50540094979132, 4601613)]}, {'query_id': '118440', 'scores': [(20.807841756611904, 7035722), (20.807841756611904, 1569786), (19.961749392044034, 8319567), (19.63093474397435, 4737856), (19.63093474397435, 1569787), (18.981466520683345, 8412702), (17.544938009402085, 6750552), (17.544938009402085, 6750548), (17.17252187241006, 6750547), (17.17252187241006, 2975214)]}, {'query_id': '119821', 'scores': [(23.018331525790302, 7093228), (21.827340546625507, 7172227), (21.827340546625507, 6527223), (21.05000934011354, 8012784), (21.05000934011354, 6267963), (21.05000934011354, 3365586), (21.05000934011354, 259724), (18.248940574181415, 8012780), (17.834191924768202, 2212944), (17.058792275865237, 6072161)]}, {'query_id': '121171', 'scores': [(21.7806784879984, 8260192), (18.85985278250794, 7165126), (18.85985278250794, 6930941), (18.85985278250794, 2337570), (16.16424828179059, 451171), (16.081212759795097, 8643275), (16.081212759795097, 451173), (15.82113923807334, 2550811), (15.82113923807334, 451172), (15.82113923807334, 259437)]}, {'query_id': '125659', 'scores': [(23.798640230794973, 6976559), (23.798640230794973, 4522348), (23.798640230794973, 1335471), (23.009272018113776, 829647), (22.840613873289826, 5881859), (22.529650637991175, 8290885), (22.529650637991175, 6635625), (22.529650637991175, 2745708), (22.529650637991175, 1106134), (21.740282425309978, 5693282)]}, {'query_id': '135802', 'scores': [(24.212925337728976, 8692623), (23.557533898012526, 8692626), (23.407148546706377, 8692628), (22.751757106989928, 8692619), (22.751757106989928, 6037338), (22.751757106989928, 6037331), (20.706323714394102, 8731214), (20.706323714394102, 7327478), (20.706323714394102, 5394718), (19.548365710972885, 8692624)]}, {'query_id': '141630', 'scores': [(40.65497717140021, 2473236), (38.86831859527273, 8319991), (38.770684055475016, 6290795), (37.88490925864993, 8319989), (37.64056963541792, 7985119), (37.510034379121095, 2366121), (37.38276492170364, 8319987), (37.294229413961865, 3500851), (37.111127493255644, 3500859), (36.91903383119926, 6496223)]}, {'query_id': '144862', 'scores': [(25.695940633888554, 92239), (25.600905731998598, 92237), (25.363787085846038, 92238), (25.29801130099314, 2865950), (25.045871686807246, 92233), (24.6143903306536, 5292762), (24.537855646036185, 6150954), (24.355228395776308, 8320087), (24.355228395776308, 7245936), (24.355228395776308, 92241)]}, {'query_id': '156498', 'scores': [(33.83615097903199, 4273410), (28.980756071320364, 5606985), (28.48960267328584, 8823842), (28.107039922246887, 3288004), (27.941711660497795, 5043014), (27.838842313294414, 7075146), (27.838842313294414, 3397582), (27.702680992441234, 3397578), (27.669779303055428, 3301559), (27.556165022331598, 3288005)]}, {'query_id': '166046', 'scores': [(30.983227422629792, 7947463), (30.983227422629792, 1076792), (30.983227422629792, 1076791), (30.242829285727453, 1076793), (28.14863777999792, 1076795), (28.14863777999792, 1076794), (25.32522428404817, 96920), (25.053632338445553, 7947466), (24.566443569476203, 96927), (24.37470165529678, 7865285)]}, {'query_id': '169208', 'scores': [(25.204019849539236, 4859433), (24.724813116269686, 4859432), (24.68243180040259, 4580137), (24.611275982230506, 4762140), (24.40556282083827, 7251639), (24.2867403489668, 4202275), (24.222755686792926, 1395417), (24.125985260433698, 85987), (23.99617407535326, 2888366), (23.992954817764154, 4762142)]}, {'query_id': '174463', 'scores': [(24.55258755738018, 8304456), (24.231330447206865, 4501665), (23.39947873976479, 102528), (21.767441165292652, 6294949), (21.675451423898956, 7089630), (21.675451423898956, 7089629), (21.22812176341202, 8304458), (21.053335409521388, 7402420), (20.003942266863135, 7611683), (20.003942266863135, 5793989)]}, {'query_id': '175920', 'scores': [(50.054641771875986, 8234235), (43.891150620497115, 101345), (39.158992979329824, 8537849), (39.06272661530688, 7615731), (39.06272661530688, 5673665), (38.58344432983391, 8537850), (38.19433162782845, 7360509), (38.19433162782845, 6815325), (38.19433162782845, 5717848), (38.19433162782845, 5673664)]}, {'query_id': '177604', 'scores': [(26.898029769775345, 8451995), (26.110126189805904, 8451992), (23.683750855733816, 5627757), (23.673914329138274, 2226834), (23.55288641976168, 5034647), (22.796060516442385, 3444466), (22.796060516442385, 1308162), (21.512377549550436, 8451989), (21.395914446677185, 8451987), (21.210926256417004, 8451991)]}, {'query_id': '181626', 'scores': [(22.17998269427472, 43641), (21.789374683128194, 436351), (21.572277717744385, 7795335), (21.572277717744385, 4798089), (21.572277717744385, 2183331), (21.41179343214739, 71074), (21.175229164501417, 8069297), (21.05910753185426, 71073), (21.021185421000865, 5427354), (21.021185421000865, 4091013)]}, {'query_id': '197312', 'scores': [(22.510948286643774, 2317108), (22.510948286643774, 2317106), (22.26049698047209, 5823013), (21.88346622348719, 5823008), (21.84886763726748, 1179432), (21.649153247366392, 5823015), (21.649153247366392, 1179433), (21.62064766680036, 1665689), (21.508506056264558, 2317100), (21.508506056264558, 1239571)]}, {'query_id': '206106', 'scores': [(32.29102125814695, 6479522), (31.509471106558244, 75400), (31.192603399299177, 7079997), (30.88476257357937, 1390597), (30.73572437953029, 5093731), (30.539732884620754, 8427974), (30.40522823746061, 8737574), (30.299612587932707, 3217609), (30.299612587932707, 2947646), (30.069423137114338, 7118029)]}, {'query_id': '227873', 'scores': [(31.899683996185864, 7276796), (31.17838055137169, 8453473), (30.56089975792402, 7276798), (30.56089975792402, 3250369), (30.56089975792402, 2054488), (28.766002403432232, 7019128), (28.406332052498705, 3324178), (25.994068261564976, 1900788), (24.621556462492308, 8453475), (24.50177155842976, 8453476)]}, {'query_id': '23849', 'scores': [(22.860597789108628, 4834498), (22.27352735354031, 2674124), (22.173121673120644, 6728787), (22.126879823743685, 4348282), (22.063188219884857, 5888570), (21.820002096060247, 2373852), (21.562268941217155, 641196), (21.539809388175367, 8246990), (21.539809388175367, 2017213), (21.40986282334399, 5363400)]}, {'query_id': '240158', 'scores': [(31.741475719500567, 2043788), (30.843099851886763, 991184), (30.830440420173588, 6536692), (30.485525404931302, 1567595), (30.252015740801763, 4874198), (27.89123712554475, 2819781), (27.84293556204974, 2819779), (27.410288428741403, 3542898), (27.11271830375189, 2819784), (27.0799611296849, 6239283)]}, {'query_id': '245052', 'scores': [(29.981277797819015, 3828386), (26.70745007120629, 5023796), (26.281694150742826, 4205737), (26.281694150742826, 66616), (26.251352001737093, 5878365), (26.032122410503554, 5666135), (26.01914731175632, 8282910), (25.933530914990836, 5143412), (25.812627073826402, 8279354), (25.812627073826402, 3379257)]}, {'query_id': '246883', 'scores': [(31.947165065529035, 3050794), (31.46147942349078, 8699192), (30.549467905360657, 5050393), (29.304231348419425, 8699195), (28.709834806392166, 4959448), (28.367187711360636, 5683913), (28.367187711360636, 2413876), (28.138464770189216, 6242496), (27.901853056598764, 8420183), (27.896254828566214, 6083923)]}, {'query_id': '253749', 'scores': [(24.719735608276025, 6190136), (24.186079731167865, 3516374), (23.944970631298283, 4070966), (22.252642351081835, 3516370), (21.92961528859971, 3516368), (21.064756127693425, 8254833), (20.254665864618588, 8254894), (19.56705236583631, 8521273), (19.51057179960666, 6260689), (19.21936747620524, 6447304)]}, {'query_id': '256942', 'scores': [(33.64069705355164, 2602557), (28.428883286631283, 2364119), (28.000785367134856, 7213042), (27.856222778788183, 4890686), (27.6743279167385, 8327014), (27.614299770651535, 3262199), (27.491327689708044, 5297913), (27.264860564033487, 5083782), (27.264860564033487, 4604331), (27.264860564033487, 1183278)]}, {'query_id': '257119', 'scores': [(30.51685014383154, 8499023), (29.86652093149418, 4363515), (27.52812150069832, 4363509), (26.007957264748203, 8032136), (25.444009205490442, 3818024), (25.372574632247918, 889398), (25.165902933586842, 8276379), (25.07003459259692, 2112755), (25.062662236984956, 5853762), (25.040978672178568, 4212490)]}, {'query_id': '258062', 'scores': [(36.51788452841922, 8694161), (36.27712417935941, 4159145), (35.63537694391206, 18103), (34.672052052926176, 3683402), (33.82109682576717, 7918982), (33.46122820737522, 2707010), (33.43838746068586, 4174313), (33.01889781798552, 3195767), (32.6221264879278, 91493), (32.41766764840526, 4174307)]}, {'query_id': '26703', 'scores': [(20.395792430012904, 3532417), (19.417386200163985, 3532415), (19.417386200163985, 1485077), (16.537761855156525, 380356), (16.003728516849062, 7265225), (16.003728516849062, 1236259), (15.97302594746483, 3733548), (15.665774569613554, 3921274), (15.618295759635853, 3132961), (15.618295759635853, 524170)]}, {'query_id': '273695', 'scores': [(31.416475771166045, 8282608), (30.81476854208236, 5542707), (30.329022199140624, 7381998), (30.270252777816594, 5542705), (29.981755214097827, 2714742), (29.699236541417115, 7381994), (29.37922875212662, 5719948), (29.37922875212662, 4171983), (29.37922875212662, 413461), (29.351969556374318, 413460)]}, {'query_id': '302846', 'scores': [(36.60916059130037, 8418433), (35.450793988797166, 8418435), (33.60031560537268, 447099), (33.362508274114205, 8283558), (33.292011336382025, 6263014), (33.21030622368226, 3892531), (33.18808682468977, 3566357), (33.079640512876765, 5696466), (32.92197194703096, 447101), (32.757052392792374, 4234633)]}, {'query_id': '318362', 'scores': [(29.383983202945302, 5654804), (28.72985848763762, 429301), (28.604555027130672, 3145979), (27.87838537775572, 4299889), (27.445150297579758, 1714531), (26.977015274214843, 1714527), (25.843795764722884, 4299887), (25.68945192903773, 543937), (25.16392818442806, 519111), (24.903429412964027, 3996891)]}, {'query_id': '324585', 'scores': [(31.039755853781863, 1051441), (28.47199444235713, 8290314), (27.261079373026295, 8290310), (26.780027247391764, 3608325), (26.142297466470826, 7977807), (25.74235121954372, 7941125), (24.729877229439474, 7359510), (24.510437749852315, 8310496), (24.459079176254214, 405282), (24.394097335190505, 6393239)]}, {'query_id': '330501', 'scores': [(28.958924433613813, 6006326), (28.44830037274712, 3718639), (28.44830037274712, 1750071), (26.427507273524906, 6647777), (26.31067911457542, 7354312), (26.31067911457542, 4397700), (26.31067911457542, 452513), (26.152101866227834, 1899109), (26.152101866227834, 294036), (25.61367643802586, 2169021)]}, {'query_id': '330975', 'scores': [(41.088286607433396, 7908218), (37.610280588215254, 5750875), (37.53937716809098, 233948), (37.49374489168959, 4108178), (37.29194981713199, 8327008), (37.263074404068774, 3297858), (37.25454564292699, 233945), (37.065790633347206, 3297857), (36.87522912505494, 1097861), (36.8429839387353, 8541461)]}, {'query_id': '332593', 'scores': [(39.211272538763744, 8434873), (38.73258522332392, 5241323), (37.93113869646589, 5241322), (37.12050580237126, 7073995), (36.49184923831097, 4431502), (36.49184923831097, 1650687), (36.14482099967, 8434869), (36.14482099967, 7073996), (35.56261912674753, 8434872), (35.11561756811462, 4614346)]}, {'query_id': '336901', 'scores': [(32.94450233763519, 3646550), (32.236238437667744, 2909600), (31.809790950654445, 8528674), (31.62490815321239, 8130980), (31.26004644535117, 7836261), (31.012519561890457, 8528681), (31.012519561890457, 8528680), (31.012519561890457, 8528679), (31.012519561890457, 8528678), (31.012519561890457, 8528677)]}, {'query_id': '3505', 'scores': [(22.010006573343226, 8285655), (21.950573672823705, 8285658), (21.821269440561743, 4520880), (21.803975274187355, 5469617), (21.724134375737677, 8285654), (21.724134375737677, 8285653), (21.715678058671784, 8285656), (21.671679253601067, 502056), (21.6687362441714, 1030851), (21.613712573904724, 1076196)]}, {'query_id': '360721', 'scores': [(29.756893699578637, 6318492), (27.003000190924112, 4284999), (26.993029713640176, 101662), (26.604899540437344, 7043084), (26.187873681420328, 7826496), (26.183977570882412, 101666), (26.02447904676214, 7826493), (25.803764067936655, 1750204), (25.575191837905404, 7826491), (25.502795635909095, 6206175)]}, {'query_id': '384356', 'scores': [(32.796880843382965, 7565188), (25.666467207721283, 2032126), (25.42274513790461, 2032127), (25.17235320773267, 3705729), (25.115843325480615, 5381456), (25.09716880488639, 6863040), (25.08750686610932, 4648730), (25.08750686610932, 412129), (25.069572621495205, 6543899), (25.069572621495205, 2032125)]}, {'query_id': '390360', 'scores': [(29.20812774772065, 604111), (27.170007397635107, 5423359), (26.787263841300792, 2203877), (26.787263841300792, 2203875), (26.44333262498268, 7679050), (25.962586354064108, 6179316), (25.366885925761537, 4683960), (25.366885925761537, 324991), (25.149788960377727, 324994), (25.149788960377727, 324992)]}, {'query_id': '405163', 'scores': [(21.633787859669223, 851536), (20.9963558838136, 6289824), (20.47542125716602, 7622492), (20.29594817067213, 429247), (20.29594817067213, 429246), (20.29594817067213, 429245), (19.137581568168926, 3293026), (19.137581568168926, 2748170), (19.137581568168926, 1518541), (19.137581568168926, 1352548)]}, {'query_id': '42255', 'scores': [(44.796906050631826, 8746197), (42.03389089148414, 8746189), (39.58768618438535, 8746188), (38.51257724374411, 8746194), (36.98244931879543, 3723425), (36.95837469015282, 3063186), (36.76714457413191, 2882385), (36.69649631230102, 7123389), (36.66907950442223, 8746196), (36.658266924874205, 6784918)]}, {'query_id': '425632', 'scores': [(30.05978476807752, 8454642), (30.05978476807752, 8454639), (28.948857275189894, 8454636), (27.69793402030919, 8454635), (27.69793402030919, 650345), (21.399799199476647, 8454641), (18.107522399557162, 8454643), (18.107522399557162, 8454640), (18.107522399557162, 8292459), (18.107522399557162, 6843069)]}, {'query_id': '426175', 'scores': [(19.99698658838695, 330964), (19.563188175308206, 6209130), (19.563188175308206, 4362255), (19.440465475504972, 6758532), (19.440465475504972, 5030738), (18.971887894438083, 6049564), (18.971887894438083, 3017346), (18.504364527577383, 6016624), (18.432005029295798, 3870458), (18.432005029295798, 3562170)]}, {'query_id': '42752', 'scores': [(21.423736085444553, 4303156), (21.377693905347037, 1254236), (21.351903583919572, 8660833), (21.351903583919572, 5932129), (21.346392079666288, 7920885), (21.1993968207006, 5325520), (21.146659474456193, 3818076), (21.13845897834657, 4821761), (21.009457638017583, 4747962), (20.99866349380827, 1986201)]}, {'query_id': '435548', 'scores': [(18.71216751526859, 8547993), (18.71216751526859, 8547992), (18.687453168331672, 7875423), (18.38619194287303, 7875421), (18.10572205791035, 7080768), (18.10572205791035, 235984), (18.075033441949994, 7875428), (18.075033441949994, 6064445), (18.075033441949994, 1394884), (17.49330233152867, 6424374)]}, {'query_id': '436707', 'scores': [(21.565781385907616, 1595287), (21.20211811043217, 2203550), (21.20211811043217, 2101427), (21.20211811043217, 1825031), (20.90723461007642, 5413961), (20.90723461007642, 3370495), (20.467506137719894, 5044105), (20.242726122362434, 8641216), (20.242726122362434, 8200621), (20.135352753267497, 6045618)]}, {'query_id': '444389', 'scores': [(23.45819939258872, 5896071), (22.97346869566392, 8495210), (19.549071666861824, 8044142), (19.549071666861824, 7705876), (19.549071666861824, 5208202), (19.272489486756484, 1722558), (19.122907863739485, 7484162), (19.122907863739485, 7150173), (19.122907863739485, 1048029), (18.88267071162128, 5251228)]}, {'query_id': '449367', 'scores': [(24.543231460340586, 8432230), (24.480981511830166, 635517), (24.166167515625446, 6632652), (24.026111679597417, 6632645), (24.026111679597417, 6632644), (23.616936435724867, 8600273), (23.11454875078192, 5717147), (22.967254871533033, 2596719), (22.941431198449777, 386955), (22.807210077816293, 1998108)]}, {'query_id': '452915', 'scores': [(26.47763572856643, 135840), (25.66301706956659, 2447912), (25.659457919932255, 3332642), (25.471092251775115, 1044828), (25.308047454635002, 953667), (25.202289231448223, 1364570), (24.98866382648245, 8366603), (24.98866382648245, 8366600), (24.98866382648245, 3779965), (24.9812473607547, 7261641)]}, {'query_id': '463271', 'scores': [(29.096906825772702, 6239584), (28.605932870375042, 8604216), (28.605932870375042, 8326637), (28.605932870375042, 6239587), (28.605932870375042, 3914186), (28.13407918316062, 2455683), (27.962043990582742, 8326635), (27.874478361005238, 8435508), (27.874478361005238, 3914192), (27.690099562680878, 6005616)]}, {'query_id': '469589', 'scores': [(34.36907003947154, 8581474), (32.451013250924284, 8581470), (32.451013250924284, 8581468), (30.51104835490213, 2210168), (28.98606515801778, 4871853), (28.65456929237459, 2210175), (27.148862851847483, 8581475), (27.148862851847483, 8581466), (27.148862851847483, 2210172), (26.786583171555897, 6441572)]}, {'query_id': '47210', 'scores': [(39.40735838251883, 2042918), (38.92777190973074, 6180504), (38.536591402233675, 2338385), (37.31869673715809, 5207329), (37.306902287019604, 2545998), (36.625264575683616, 4748997), (36.017626027731254, 3858013), (35.860807168286804, 2546002), (35.860807168286804, 2546001), (35.8316351751634, 7367873)]}, {'query_id': '482726', 'scores': [(13.859663652421483, 8730812), (13.66661662883713, 8572751), (13.377394479799944, 6318274), (13.377394479799944, 4734241), (13.377394479799944, 22062), (13.367067288601746, 1272524), (13.285759535328626, 4734239), (13.214007644846369, 4768911), (13.214007644846369, 3512937), (13.214007644846369, 1985718)]}, {'query_id': '48792', 'scores': [(32.86343825946438, 64048), (32.54030528565261, 64047), (29.789532277819003, 64041), (20.073607219357562, 6619302), (19.72312698412783, 5803986), (19.72312698412783, 3820076), (19.541548782241556, 3820082), (19.5062156411961, 6152219), (19.504387254247433, 6152218), (19.504387254247433, 3820080)]}, {'query_id': '50122', 'scores': [(28.678499136309696, 8273511), (27.08316087286957, 1417366), (26.685286289400587, 8273514), (26.437411560413253, 8273517), (26.437411560413253, 425197), (24.798201767254916, 7341000), (21.40532983806166, 8361668), (21.270651878017254, 8337928), (21.270651878017254, 5999140), (20.91664882426648, 8273519)]}, {'query_id': '514096', 'scores': [(14.269459064819728, 4786295), (14.120858133037931, 6286608), (14.120153497916533, 6474925), (14.120153497916533, 6285508), (14.040509954807074, 1837337), (14.004171081505467, 3534826), (13.92382290327461, 8460993), (13.92382290327461, 5936453), (13.911465910981759, 7016672), (13.911465910981759, 3708394)]}, {'query_id': '519025', 'scores': [(18.300780777498236, 4232478), (18.233437569483172, 288148), (17.898975285501816, 2692126), (17.80945291058821, 6805536), (17.80945291058821, 6526749), (17.665674550878474, 6526750), (17.665674550878474, 3394216), (17.665674550878474, 1697202), (17.649222141983184, 6236703), (17.649222141983184, 5931056)]}, {'query_id': '53233', 'scores': [(31.724497861247475, 8333467), (30.782396091072492, 8257054), (30.57942593792626, 8333470), (30.57942593792626, 8333463), (30.19995999943549, 8257048), (30.19995999943549, 5256809), (29.23670030942045, 8257051), (29.23670030942045, 7355044), (29.23670030942045, 5516316), (29.23670030942045, 579087)]}, {'query_id': '537060', 'scores': [(25.412451467483137, 8420027), (25.375496254269283, 8420024), (25.012797561641428, 8420032), (25.012797561641428, 8420025), (23.998291492966317, 8420030), (23.998291492966317, 8420029), (23.998291492966317, 8420026), (23.998291492966317, 7386519), (23.998291492966317, 1086305), (23.1412116039733, 8420031)]}, {'query_id': '537817', 'scores': [(28.071834693812697, 1956211), (27.99187457017451, 6665873), (27.937850821796612, 8527917), (27.837353060738288, 540370), (27.823947882452188, 5264609), (27.746871090648984, 1956207), (27.376680640698915, 3759425), (27.353119364751954, 540368), (27.327665861049645, 2025709), (27.222159131262693, 1095538)]}, {'query_id': '543273', 'scores': [(30.8387550403827, 8507214), (30.16653291555845, 8507211), (28.698847922063155, 8507209), (28.456521921370953, 4308075), (27.78928550256225, 5990522), (27.211308317433822, 8507212), (26.895980091042137, 8507208), (26.895980091042137, 4308073), (26.895980091042137, 3143117), (26.095531516355546, 4067764)]}, {'query_id': '545355', 'scores': [(35.63022198335884, 4290924), (34.50511598461808, 4290920), (33.790504015447745, 8547218), (33.790504015447745, 8547215), (33.62184008040103, 8547219), (33.62184008040103, 4290921), (33.306511854009344, 4290922), (30.263901368617933, 4290923), (26.369290873392224, 4290918), (26.369290873392224, 3294315)]}, {'query_id': '555530', 'scores': [(27.379980506977045, 1897716), (26.89954792132373, 7475084), (26.834614268454494, 4296807), (26.482235115237067, 4137963), (26.40644589630039, 1754575), (26.380465719765265, 6485976), (26.066555328310393, 6150835), (26.058963918578577, 3948935), (25.836781873969752, 4238674), (25.674586359413343, 1483513)]}, {'query_id': '583468', 'scores': [(20.353395278900887, 7491704), (20.250393351859575, 8326078), (20.170026681747185, 7491700), (20.170026681747185, 1675562), (20.09663471522896, 7706119), (20.09663471522896, 2267054), (20.09663471522896, 2267052), (20.09663471522896, 1675564), (20.005090260541806, 6654310), (20.005090260541806, 2080306)]}, {'query_id': '586148', 'scores': [(23.233863939573926, 5056079), (22.515785037124484, 960743), (22.45176908654177, 8271038), (22.405142397873863, 6148926), (21.985929369464728, 7779366), (21.898998477293198, 6854043), (21.898998477293198, 1658943), (21.875286730214103, 2811037), (21.848540970336344, 8567688), (21.816326381767414, 5678812)]}, {'query_id': '590019', 'scores': [(22.028152614086693, 4408763), (21.960541219746414, 2831848), (21.957367279684515, 2831847), (21.925466825791865, 632727), (21.6468705147638, 5075460), (21.497372467317575, 3453506), (21.468076822950728, 3266922), (21.440115200181243, 1248322), (21.438139494170564, 6066327), (21.430754974647755, 8369826)]}, {'query_id': '605127', 'scores': [(29.54948404811826, 1137979), (29.29054441109448, 8642036), (28.217137581717346, 8594907), (28.217137581717346, 1137978), (27.90405594351246, 5121398), (26.968448798792522, 3679424), (26.531213568429052, 8547782), (26.270135721135908, 8642038), (26.270135721135908, 1137982), (25.48261988704425, 6888324)]}, {'query_id': '610265', 'scores': [(31.78958420887561, 474853), (31.477248824789537, 222769), (31.46975308639441, 2877322), (31.46975308639441, 683818), (31.362871495751147, 2877320), (31.35522339237378, 1148168), (31.30033911772799, 4758657), (31.29245936379738, 1148169), (31.2644875539485, 7009719), (31.23099309757871, 2039693)]}, {'query_id': '611953', 'scores': [(41.95952831774613, 484756), (36.60579472237875, 7197690), (33.34764124114653, 3562872), (33.34764124114653, 484758), (32.84694580888977, 456354), (31.697129061277842, 8062950), (31.67903804615015, 456353), (31.40394501555815, 456359), (31.313295934986996, 8062949), (30.86999404774277, 8371935)]}, {'query_id': '640502', 'scores': [(24.137307504544477, 3551380), (23.979533107070864, 5827712), (23.7259897108563, 5827713), (23.245176778833656, 6730887), (23.245176778833656, 3271456), (23.245176778833656, 1861075), (23.24228895824674, 6985038), (23.218890447598476, 5150987), (23.127463852892276, 3678130), (23.071816021131035, 77134)]}, {'query_id': '64647', 'scores': [(27.09904634719087, 8293649), (26.778668501629156, 8293652), (25.15409869298266, 4161653), (23.884000404013776, 8293648), (20.012602224013186, 8599633), (19.33726995474058, 1422031), (19.31166950715128, 6508277), (19.12412981749385, 7509908), (19.12412981749385, 5692182), (19.12412981749385, 1310940)]}, {'query_id': '653399', 'scores': [(23.014136012978, 1969533), (21.896634527665178, 7133996), (21.532149073346694, 8458007), (21.436484406288333, 7485466), (21.436484406288333, 6343750), (21.436484406288333, 3111946), (21.436484406288333, 2007733), (21.12100887379972, 1674561), (20.9902792149612, 2399252), (20.965139502650423, 7092040)]}, {'query_id': '655526', 'scores': [(24.431139065384585, 6040076), (23.742330237490634, 8425101), (22.977834387595756, 206999), (21.00283059470325, 6707899), (21.00283059470325, 3864910), (21.00283059470325, 3021665), (21.00283059470325, 2105608), (18.570004216190537, 1826238), (18.310887878290202, 5883282), (17.894731335601787, 7135528)]}, {'query_id': '655914', 'scores': [(19.661471306404803, 8613490), (19.067925850907507, 1270195), (17.56123354079898, 8613495), (17.56123354079898, 8058173), (17.56123354079898, 5866957), (17.56123354079898, 5235329), (17.56123354079898, 2789189), (14.473927734054586, 3209412), (14.399575365557732, 6248270), (14.399575365557732, 2519335)]}, {'query_id': '660198', 'scores': [(19.106625083618113, 2845619), (18.248321816491032, 523901), (18.228771270922223, 2410610), (18.08250027555718, 4434042), (18.032999183188267, 7327517), (18.032999183188267, 4264632), (18.032999183188267, 3556727), (17.784910062208994, 7327515), (17.784910062208994, 4264630), (17.784910062208994, 3604202)]}, {'query_id': '67316', 'scores': [(37.90866063178886, 6680964), (33.769282667720894, 7630885), (33.20283994442197, 6764671), (32.52083576535737, 6179107), (32.397210962163044, 4163705), (32.129180411721094, 7112302), (32.10955148724639, 4252546), (32.10955148724639, 3810719), (31.581609540089765, 8431637), (31.458095581147816, 1974375)]}, {'query_id': '673670', 'scores': [(17.282007517506695, 3607501), (17.118969710737762, 7257303), (17.118969710737762, 2240359), (16.880100365936773, 8008345), (16.880100365936773, 3607497), (16.880100365936773, 1905068), (16.880100365936773, 1398756), (16.496461721256388, 2240358), (16.496461721256388, 2240357), (16.496461721256388, 2240355)]}, {'query_id': '701453', 'scores': [(24.206111022723093, 4355647), (24.206111022723093, 4330679), (23.84573194781223, 4355649), (23.754391721719188, 8626149), (23.567346797193544, 6999087), (23.457943162093635, 4330677), (23.233932527084256, 2036081), (23.186607907625962, 4355645), (23.126317639001012, 2036076), (23.046887602558613, 4330673)]}, {'query_id': '703782', 'scores': [(21.86505116274649, 4061595), (21.488922667992632, 4061598), (21.05555722838493, 4061597), (21.05555722838493, 344218), (21.05276311422473, 185599), (20.759962887052026, 4647484), (20.70567598019244, 4948640), (20.634155134647266, 3490915), (20.634155134647266, 611402), (20.53564529078316, 7687244)]}, {'query_id': '708979', 'scores': [(20.3801296232292, 6157147), (20.340027978512836, 4805197), (20.293813233439856, 279265), (20.254024012882986, 3811546), (20.23493691437428, 2889574), (20.1705682596444, 6313188), (20.157369700367248, 1447346), (20.134207230448848, 8088373), (20.11169194057883, 6409763), (20.11169194057883, 3422810)]}, {'query_id': '716113', 'scores': [(22.727449504139592, 8520612), (22.727449504139592, 8520611), (22.41032230175625, 8520617), (20.94877954294606, 8520618), (20.94877954294606, 8520614), (20.94877954294606, 8520613), (18.53161267260613, 8520616), (18.53161267260613, 8520610)]}, {'query_id': '730539', 'scores': [(28.021698298220315, 8577877), (26.802454475047874, 8276218), (26.368731031343962, 2666444), (26.117734758151407, 2666436), (25.328818562596783, 8577880), (23.326185143111964, 8577876), (23.326185143111964, 5428552), (23.326185143111964, 4605022), (18.145072497756377, 8276217), (18.145072497756377, 7916064)]}, {'query_id': '735482', 'scores': [(20.95904057147796, 8405801), (20.704858530079424, 8405802), (20.419876258588182, 4823368), (20.241858271284883, 8405807), (20.241858271284883, 338960), (20.02617748777707, 8405806), (19.57256803445122, 8405805), (18.465664950457082, 4117288), (18.421097883700984, 8405804), (18.421097883700984, 8405803)]}, {'query_id': '735922', 'scores': [(20.902884347726427, 5537112), (20.902884347726427, 5537109), (20.129240188307204, 5854095), (20.129240188307204, 281701), (20.129240188307204, 281697), (20.129240188307204, 180247), (19.569426147540504, 8626887), (19.264657236254145, 7371194), (19.264657236254145, 5861444), (19.264657236254145, 1432224)]}, {'query_id': '75198', 'scores': [(24.390280548779728, 7797928), (24.181549528186487, 2579716), (23.778621801776985, 8256960), (22.848041523603513, 6862216), (22.318185855943753, 6862217), (22.318185855943753, 3916717), (22.318185855943753, 2140526), (22.318185855943753, 2140522), (21.04990302096848, 7088445), (21.04990302096848, 2932859)]}, {'query_id': '768208', 'scores': [(20.624078671734345, 52162), (20.33630082980317, 52155), (19.874112174580368, 8525681), (19.874112174580368, 52159), (19.010020340902962, 3515884), (19.010020340902962, 3515882), (19.010020340902962, 813102), (19.010020340902962, 52160), (19.010020340902962, 52158), (19.010020340902962, 52157)]}, {'query_id': '779302', 'scores': [(26.110665801292683, 1583173), (25.545699157192303, 8768560), (25.545699157192303, 8768554), (24.111553060629795, 8768563), (24.111553060629795, 8768562), (24.111553060629795, 8768561), (24.111553060629795, 8768558), (24.111553060629795, 8768555), (20.048992815996883, 2411546), (19.742264120271315, 4638389)]}, {'query_id': '792635', 'scores': [(21.020247038837134, 4294317), (20.10632325453987, 8682263), (20.10632325453987, 8682259), (20.10632325453987, 8682258), (20.10632325453987, 8682254), (20.10632325453987, 6306494), (20.10632325453987, 6306491), (20.10632325453987, 4294318), (20.10632325453987, 4294312), (20.10632325453987, 4294311)]}, {'query_id': '794223', 'scores': [(19.43874145001271, 7790213), (19.167779422391188, 2519053), (18.42471838424802, 8445687), (17.234891954328624, 471681), (17.195809744242013, 6209396), (17.073661729071144, 5946388), (17.073661729071144, 892194), (16.330600690927977, 4226039), (16.330600690927977, 3798578), (16.330600690927977, 3293492)]}, {'query_id': '794429', 'scores': [(25.42923530959834, 1208450), (24.305882164771873, 1208449), (24.13566172070159, 64014), (23.009588013775137, 8618468), (23.009588013775137, 4652685), (23.009588013775137, 719934), (19.820298316040052, 6098420), (19.816250855773056, 1246185), (19.7502210958559, 3926405), (19.6078896007601, 4627633)]}, {'query_id': '801118', 'scores': [(25.97279122372523, 2932991), (25.487500501928423, 4799238), (25.38640740241662, 7615528), (25.38640740241662, 3975963), (24.59157977764139, 1930307), (24.503542617041948, 3263703), (24.147630496117202, 7486725), (23.90534586247828, 4214955), (23.873215102259547, 3360132), (23.86042062132546, 5219693)]}, {'query_id': '804066', 'scores': [(15.725771106763728, 73387), (15.555198781648528, 6152070), (14.729165959101302, 8567803), (14.729165959101302, 7922526), (14.729165959101302, 7621407), (14.729165959101302, 5655169), (14.729165959101302, 1504069), (14.729165959101302, 1259677), (14.363198580400871, 7427197), (14.363198580400871, 3298496)]}, {'query_id': '808400', 'scores': [(22.135317215125845, 3691917), (21.36585660255728, 3691923), (20.756014685824034, 3377142), (19.927255157955777, 7282136), (19.927255157955777, 7252539), (19.927255157955777, 4576228), (19.927255157955777, 1995351), (18.53094378838643, 5834423), (18.421902682259145, 5598346), (18.109785975014013, 5834421)]}, {'query_id': '809525', 'scores': [(27.42538122228916, 3793929), (24.573920821126393, 2077416), (24.020395911137925, 1724922), (23.50350983560186, 1428570), (23.20743837211966, 3793922), (23.20743837211966, 1428575), (22.653913462131193, 7059360), (22.653913462131193, 1428574), (22.653913462131193, 1428573), (22.615294216551817, 2077420)]}, {'query_id': '814183', 'scores': [(31.153893624466157, 8679359), (24.38831177251556, 8444060), (22.60609281146659, 8679357), (22.03700036792222, 7274881), (20.12821248945201, 510865), (20.12821248945201, 510863), (19.67075311469174, 510861), (18.81550297927036, 8552189), (18.81550297927036, 7013314), (18.81550297927036, 878335)]}, {'query_id': '819983', 'scores': [(15.629884651305396, 8145644), (15.629884651305396, 1338142), (15.57585673406358, 1617043), (15.539899738217027, 4580581), (15.538363451105887, 8312879), (15.536364709319958, 8328822), (15.507709766537394, 8439123), (15.501440923522967, 8831812), (15.501440923522967, 8045345), (15.50007042902299, 1221184)]}, {'query_id': '849550', 'scores': [(22.22713073927145, 1373164), (22.165456766370884, 7274031), (21.96997360322762, 6758594), (21.60791069780627, 8582102), (21.557392223190995, 7767168), (21.557392223190995, 4061418), (21.339107677479376, 8000555), (21.339107677479376, 7767169), (21.339107677479376, 1419091), (21.330688592998968, 5690724)]}, {'query_id': '85020', 'scores': [(31.134208105931577, 3152840), (30.938365088366425, 198481), (30.649419274089674, 198476), (30.42571287876232, 6290170), (30.08813157514495, 6974021), (29.939957010168982, 1770218), (29.20986148918064, 603541), (29.14034554155876, 147291), (29.028698716590277, 1640375), (29.001321691242758, 1777604)]}, {'query_id': '850358', 'scores': [(29.165753578345395, 2852825), (29.165753578345395, 2852824), (28.56096101171795, 2852830), (28.00875910305811, 6427506), (27.478041568035604, 2852827), (27.442725295355626, 2852826), (24.776979206551403, 8037013), (22.871289688750835, 389081), (22.835507535477, 2278756), (22.832537208035, 6427505)]}, {'query_id': '86606', 'scores': [(27.076564631775533, 5013458), (26.989033078409605, 8422274), (26.977449490254234, 6253490), (26.977449490254234, 6253489), (26.977449490254234, 677277), (26.861574887309366, 3077883), (26.76826153125031, 747160), (26.42624971078849, 6897895), (26.32043388675848, 677274), (26.32043388675848, 677272)]}, {'query_id': '877809', 'scores': [(30.542379035335212, 4888926), (30.33254311591184, 5940394), (30.33254311591184, 5940393), (30.10862990554012, 6704838), (29.9203524567138, 741714), (29.1166392035293, 4888929), (28.97115142078609, 7040022), (28.963444473452608, 7040017), (28.930853284762634, 1550610), (28.66188495571084, 7346771)]}, {'query_id': '883915', 'scores': [(32.02175426985656, 4042980), (31.238254073296368, 6172403), (30.849592862342558, 4042985), (30.803892593077062, 3535079), (30.644981344791262, 5406348), (30.61920331726396, 6172404), (30.124736384916716, 4857055), (29.95739605589631, 5559530), (29.91600536432348, 5406347), (29.896227397992213, 1845958)]}, {'query_id': '88495', 'scores': [(15.077352965850473, 1091557), (15.046667342820506, 2875829), (15.017734827039696, 6579243), (14.99496516252983, 6846565), (14.99496516252983, 1667678), (14.977151395198625, 7950301), (14.977151395198625, 5897696), (14.974296984491962, 5915611), (14.926849591670678, 5459872), (14.925449214907948, 3668359)]}, {'query_id': '911232', 'scores': [(48.94103136051328, 8816622), (46.601150696209956, 2999930), (45.74537563176936, 2999933), (44.351854430231576, 8816620), (42.68722004445147, 6928655), (41.57924920747766, 1879470), (39.374883445396314, 2999925), (37.829807672599586, 8816626), (37.4224271939379, 2999926), (36.80304716499916, 2999929)]}, {'query_id': '914916', 'scores': [(24.115066183987224, 6042350), (24.115066183987224, 2631739), (23.900148596496386, 2631743), (23.702185841976544, 7040507), (23.58277717125479, 8520449), (23.58277717125479, 8481901), (23.58277717125479, 5177082), (23.58277717125479, 2001628), (23.58277717125479, 2001626), (23.5243453755621, 6071696)]}, {'query_id': '91576', 'scores': [(21.626661335749745, 8279891), (21.254055200294314, 3780281), (21.186062669141794, 1382984), (20.30820885644591, 4161753), (16.997766322096204, 3540074), (16.798465365922368, 6805821), (16.798465365922368, 6137131), (16.798465365922368, 3255380), (16.42585923046694, 4066736), (16.42585923046694, 3880194)]}, {'query_id': '918162', 'scores': [(37.736907364185775, 2692389), (37.736907364185775, 2692384), (37.647986597970856, 2692388), (36.78845626874412, 267612), (36.12998838129908, 8217238), (35.36692533963837, 4658366), (35.36692533963837, 1290157), (35.36692533963837, 672942), (34.032798311938706, 2334496), (32.285989138142504, 1223048)]}, {'query_id': '938400', 'scores': [(25.88486975747914, 7239123), (24.674026489890096, 6499450), (24.674026489890096, 2969902), (22.757135958437825, 2870510), (22.49427807091705, 3074064), (21.827023433364314, 7029656), (21.827023433364314, 3424110), (21.827023433364314, 2371820), (21.827023433364314, 790219), (21.827023433364314, 47285)]}, {'query_id': '940547', 'scores': [(31.500445897881193, 4481817), (30.889070109606976, 6404591), (30.20682517048933, 4664999), (29.59174492765908, 3822464), (29.582790308462954, 4820847), (29.13328556933979, 7793152), (29.13328556933979, 4100051), (29.13328556933979, 3408094), (28.957481695636652, 1976277), (28.237017424336784, 2947470)]}, {'query_id': '945835', 'scores': [(28.60197933240492, 4431195), (27.78575375488353, 4539379), (27.670557836323006, 5548718), (27.670557836323006, 4431193), (27.43742163126144, 4425412), (27.43742163126144, 61200), (27.43742163126144, 61192), (27.157326309060814, 5548713), (27.157326309060814, 4539378), (26.861507015064547, 61193)]}, {'query_id': '978031', 'scores': [(19.435170985035278, 8744912), (19.144578836480417, 8744911), (18.710755081464356, 4755047), (18.702871910179383, 8744914), (18.14379076161408, 261672), (18.078843186159986, 8802560), (17.99327733278395, 3698979), (17.87542247813394, 5287819), (17.87542247813394, 4543384), (17.87542247813394, 2398979)]}, {'query_id': '985594', 'scores': [(19.67035606397626, 1657406), (19.39588597936264, 8677959), (19.39588597936264, 8677953), (19.39588597936264, 7740126), (19.39588597936264, 4286031), (19.39588597936264, 1657408), (19.39588597936264, 1657405), (19.39588597936264, 655890), (18.95507038892258, 7740128), (18.95507038892258, 7740125)]}, {'query_id': '99005', 'scores': [(45.2135054158107, 8257322), (44.35849888085889, 6493343), (43.32028239431068, 6493344), (43.145437455444196, 2875400), (43.145437455444196, 1375045), (41.83955027819647, 297028), (41.83955027819647, 297023), (39.153255240284125, 6327640), (39.06156508279107, 8257320), (38.66518814029489, 8476970)]}, {'query_id': '997622', 'scores': [(27.76902319322393, 4543865), (27.10310266495629, 73465), (25.929397523008838, 4518222), (25.929397523008838, 2275574), (25.710176817953887, 73459), (25.41268901627689, 4558331), (25.080989737030873, 4643397), (24.23898387432944, 8533344), (24.23898387432944, 3913148), (23.86791654571513, 5430569)]}, {'query_id': '999466', 'scores': [(22.22063876135589, 8748919), (22.22063876135589, 8748914), (20.771466668223987, 8748922), (20.771466668223987, 8748918), (18.3747589757366, 8748921), (18.3747589757366, 8748920), (18.3747589757366, 8748916), (18.3747589757366, 8748915), (18.3747589757366, 8748913)]}, {'query_id': '132622', 'scores': [(23.01604771891932, 5901873), (22.71211571786607, 962815), (22.047486249068534, 2459821), (22.047486249068534, 821354), (21.95264894655249, 2979805), (21.328911375048648, 7122663), (21.039073000281075, 5656719), (20.9665369265526, 3162996), (20.666644970146464, 6358747), (20.36034990519786, 3541644)]}]\n"
          ]
        }
      ],
      "source": [
        "print(query_processing(trec_dl_2020.queries_iter(), daat_tfidf))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SKR6d6ujrwjC",
        "outputId": "eb843cbe-2386-41a6-c8e7-9b44014fef18"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "query_processing (136249.989 ms)\n",
            "[{'query_id': '1030303', 'scores': [(37.13602502160565, 8726436), (32.82578122640052, 8726435), (32.82578122640052, 8726433), (32.82578122640052, 8726429), (30.70863607555851, 8726437), (30.70863607555851, 8726434), (30.70863607555851, 8726430), (30.70863607555851, 7156982), (20.09607619370668, 1305521), (20.09607619370668, 1305520)]}, {'query_id': '1037496', 'scores': [(27.164761525836614, 7766587), (27.164761525836614, 7766585), (27.164761525836614, 5927420), (27.164761525836614, 4760912), (25.1453728825241, 4905511), (25.1453728825241, 4760914), (25.1453728825241, 4174377), (25.1453728825241, 3725937), (18.72239393334702, 4174376), (18.296884980316406, 3725932)]}, {'query_id': '1043135', 'scores': [(36.87987445266299, 8696961), (35.933761717917605, 650642), (34.965363265314046, 3378240), (34.82276430970873, 8696958), (34.312225997811325, 4994428), (33.41670021144713, 4355523), (33.41670021144713, 2514458), (32.78034666132486, 650641), (31.436423214371146, 2664995), (30.82455348018611, 4721484)]}, {'query_id': '1045109', 'scores': [(34.19487039697934, 8838896), (30.129545984624254, 8838890), (28.860477596135542, 8838893), (28.701724152162242, 8838892), (28.153437726491497, 8838891), (26.761633722865753, 8838899), (26.761633722865753, 8838898), (26.761633722865753, 8838897), (26.761633722865753, 8838895), (21.013377074207764, 7620329)]}, {'query_id': '1049519', 'scores': [(28.56064263010628, 7105), (27.818609848717962, 5355123), (25.53010512617196, 1316185), (25.362403072758422, 7106), (24.777212305223607, 7668125), (22.65232350353714, 5355121), (22.488707582677602, 6080565), (22.0739416296838, 5355126), (22.00765545704307, 6950445), (22.00765545704307, 6580258)]}, {'query_id': '1051399', 'scores': [(29.575974242084552, 4426188), (29.575974242084552, 69818), (29.575974242084552, 69813), (29.452244366377762, 2845499), (29.019131142230236, 7848674), (28.98430649216766, 3773237), (28.68125490244961, 8712317), (28.485325955804058, 2866972), (28.25058202040101, 3446937), (28.12172342732147, 3943228)]}, {'query_id': '1056416', 'scores': [(43.48907029835839, 8739209), (41.02607316236592, 2308626), (37.62130424677382, 8757245), (37.62130424677382, 2059786), (35.565564644482905, 3790242), (35.44467447809854, 895432), (34.417165894385676, 7321435), (34.29627572800131, 2348501), (34.29627572800131, 1367298), (32.26903594513811, 2348504)]}, {'query_id': '1064670', 'scores': [(32.914793712531775, 8680901), (32.06413070969825, 8680903), (31.69183135257443, 8680905), (31.077692190131927, 8680897), (29.796618308113246, 8680904), (28.573655948155896, 8680902), (24.724207855807858, 7734392), (23.50124549585051, 3807086), (22.887222815463574, 129462), (22.887106333408013, 8337453)]}, {'query_id': '1065636', 'scores': [(32.13692641557028, 4371749), (31.25511911696255, 1062134), (31.175178692601186, 8490752), (26.858925790916082, 7411479), (26.39675824713418, 436583), (26.245829481570585, 3822992), (26.245829481570585, 597867), (26.102079769511043, 6307138), (25.83717725185279, 7411495), (25.762702017570106, 444919)]}, {'query_id': '1071750', 'scores': [(52.60972937985006, 3928913), (50.998790298666265, 8698268), (48.8128472491279, 8698276), (47.5994270983294, 8698274), (47.5994270983294, 8698269), (47.5994270983294, 3575934), (46.1699817709662, 8378390), (40.48248232507933, 8378386), (39.05303699771613, 8378389), (39.05303699771613, 8378385)]}, {'query_id': '1103153', 'scores': [(25.614751592222284, 8226438), (25.322313885119613, 8226436), (24.930172896167527, 8226445), (24.902565344478262, 7570096), (24.54923606597886, 6113506), (23.837049818234835, 8226444), (23.837049818234835, 8226440), (23.837049818234835, 8226439), (23.09470794635881, 8226443), (23.09470794635881, 8226441)]}, {'query_id': '1103791', 'scores': [(17.30903162087153, 8384424), (17.038590400484036, 8384416), (16.879445471964058, 4034845), (16.780441962841426, 8220282), (16.694099819008517, 4034852), (16.55600112945586, 1512326), (16.5439328936394, 8384417), (16.5439328936394, 8146212), (16.383198960767587, 8220278), (16.383198960767587, 8214165)]}, {'query_id': '1104501', 'scores': [(36.07442617891704, 6871582), (35.81297390744045, 6067096), (35.802183908778545, 5286080), (35.373549001613824, 2204057), (35.338146425164496, 2204054), (35.27536209957584, 2090898), (35.210441753584206, 5594847), (35.14479513430444, 7431269), (35.121782516435864, 2283306), (35.09948763109822, 3604439)]}, {'query_id': '1105792', 'scores': [(21.51770058291308, 8201616), (21.51770058291308, 8201611), (21.31470340760258, 8201618), (21.31470340760258, 8201612), (21.017288941449983, 8201613), (20.539623283689757, 8201615), (20.539623283689757, 8201610), (20.539623283689757, 6977017), (19.6465961843989, 8201614), (19.6465961843989, 7102720)]}, {'query_id': '1105860', 'scores': [(28.313045350974456, 3593906), (28.313045350974456, 3593905), (28.313045350974456, 3593904), (28.066687266498956, 8200940), (28.026995388461547, 8282567), (28.026995388461547, 8282566), (28.026995388461547, 8282565), (27.824003562189137, 7237676), (27.80393170478729, 3593907), (27.686325329248515, 4866505)]}, {'query_id': '1106928', 'scores': [(32.35964120893993, 4285175), (31.49798753258775, 2168463), (30.277382400824493, 5186636), (29.934481189841122, 7744803), (29.62139747546501, 8190627), (29.62139747546501, 8190624), (29.118554103751585, 2866124), (29.05421131544879, 8190620), (28.87455403454204, 1392371), (28.155189259162842, 6949062)]}, {'query_id': '1106979', 'scores': [(33.04043816379437, 8190140), (33.04043816379437, 8190137), (32.94120133298984, 7674419), (30.375706989944092, 7674423), (29.694403774851565, 5383176), (29.694403774851565, 4219185), (29.694403774851565, 1589841), (28.76853017485402, 8190135), (27.21073850578742, 8190134), (26.829894021844776, 348057)]}, {'query_id': '1107315', 'scores': [(21.61450985364886, 4598318), (20.507081108128187, 8187302), (19.769500198521722, 8817722), (19.769500198521722, 2696421), (19.769500198521722, 2696417), (19.769500198521722, 2696416), (19.319920793207245, 2687026), (19.319920793207245, 2687024), (19.319920793207245, 1708353), (18.675564336690158, 3430125)]}, {'query_id': '1107440', 'scores': [(33.799493738424545, 8186231), (31.856703544081675, 8186223), (25.06835933046453, 789989), (23.958479599927493, 1409567), (21.142673285174283, 3971063), (20.58385212750857, 1879424), (20.408754161530688, 8186227), (20.408754161530688, 5177729), (20.408754161530688, 5177724), (19.678793695377344, 2888123)]}, {'query_id': '1108450', 'scores': [(30.97590845311654, 1629571), (28.625692970081587, 4400204), (17.62204540010716, 1629577), (17.33705275212699, 6285212), (17.33705275212699, 6285210), (17.09514038814382, 8177145), (17.09514038814382, 4747082), (17.09514038814382, 4400207), (17.09514038814382, 4400205), (17.09514038814382, 1629573)]}, {'query_id': '1108466', 'scores': [(32.370544026766346, 8177012), (30.390105521457272, 4944265), (30.390105521457272, 3267306), (30.390105521457272, 753377), (30.13155264408124, 2702004), (29.924987560214298, 841054), (29.758691159685338, 5380573), (29.758691159685338, 3482588), (29.758691159685338, 3267303), (29.69511183471414, 2002688)]}, {'query_id': '1108473', 'scores': [(39.34059712381017, 8176948), (37.68282511102558, 1188068), (37.16912082641594, 1188069), (35.07552720572149, 8176949), (31.163909852171926, 1970473), (30.894958052764473, 3978700), (30.280919883963882, 8176947), (30.006105571426232, 1188063), (29.324416850652305, 5562098), (29.32287310250773, 357762)]}, {'query_id': '1108651', 'scores': [(28.592520796722653, 6348741), (28.40126913590069, 8175412), (26.83656060387166, 1371583), (25.879959013593783, 5027435), (24.05154746508245, 6985088), (24.05154746508245, 5997928), (23.73405119966072, 6663873), (23.64573000746448, 1325612), (23.55031615206348, 6985085), (23.55031615206348, 4617850)]}, {'query_id': '1108729', 'scores': [(34.46548470515693, 3405694), (31.260368667882965, 7789132), (29.328741387876683, 5449883), (26.673919335707872, 3405688), (25.126006579146264, 8174842), (24.446233335738857, 8322380), (24.29624851460401, 334958), (24.13167319151686, 7993178), (23.92890109383491, 1820798), (23.871601357354894, 5623921)]}, {'query_id': '1109699', 'scores': [(17.838621470487702, 2769074), (17.82912688788502, 4872045), (17.780860005807057, 8394502), (17.717270304021795, 7722707), (17.717270304021795, 7722706), (17.717270304021795, 7471054), (17.717270304021795, 2341119), (17.717270304021795, 737337), (17.63771206021638, 7744060), (17.63771206021638, 178437)]}, {'query_id': '1109707', 'scores': [(34.94151316497033, 1911564), (33.949340895762504, 1719374), (33.840084503169706, 5947139), (33.83131666002568, 4122615), (33.36118900149726, 771002), (33.25338687566345, 4122618), (33.241440091836026, 1719371), (33.05516998358234, 5740304), (32.8663855939665, 4285843), (32.7158712336693, 4099609)]}, {'query_id': '1109850', 'scores': [(33.016789385490725, 8164028), (31.88684380605354, 8164035), (31.88684380605354, 8164030), (31.044837943352107, 8164033), (25.71940524207831, 3633621), (25.71940524207831, 3633617), (23.38605687821022, 8164032), (22.22063876135589, 8164036), (18.3747589757366, 8164029), (15.321955960837355, 5382247)]}, {'query_id': '1110678', 'scores': [(26.0675417177549, 7139329), (25.007542116324885, 8155914), (25.007542116324885, 6632692), (25.007542116324885, 2342590), (25.007542116324885, 2342589), (24.534580786659845, 8155909), (24.534580786659845, 5719597), (23.254465852421404, 2714688), (23.254465852421404, 2126363), (23.254465852421404, 2096160)]}, {'query_id': '1112142', 'scores': [(44.306069518791695, 2542782), (43.8110027055872, 3651292), (43.8110027055872, 3651291), (43.8110027055872, 3651290), (43.58272649544322, 7866090), (39.649360688789024, 2238202), (37.931331524148405, 1946438), (37.23978350620301, 8240097), (37.23978350620301, 7844243), (37.23978350620301, 5670185)]}, {'query_id': '1113042', 'scores': [(27.768761791257056, 7072957), (27.537084150153333, 2410652), (26.93764472193051, 355186), (26.429509473953956, 2499270), (26.17508316216948, 3521102), (25.699416579268068, 7834150), (25.699416579268068, 4931496), (25.699416579268068, 4243566), (25.40616954106973, 2326525), (25.40616954106973, 2326519)]}, {'query_id': '1113256', 'scores': [(50.37766796768834, 7561115), (50.02546094432704, 8130535), (50.02546094432704, 7561120), (48.7793839656685, 7561117), (47.69589360057542, 7561116), (47.34017985027023, 7561114), (46.96111350052801, 8130530), (46.96111350052801, 7561113), (45.52541611207362, 7561111), (41.880953510612486, 8130528)]}, {'query_id': '1114166', 'scores': [(23.973880881167275, 6019023), (23.733030202193945, 3594677), (23.733030202193945, 3142379), (23.733030202193945, 2463783), (23.105358041182587, 8478109), (22.853308585460223, 5989751), (22.707228582310428, 5193359), (22.70631790383011, 1816877), (22.430263738038953, 5005484), (22.430263738038953, 3951381)]}, {'query_id': '1114286', 'scores': [(14.631235349876171, 6846966), (14.631235349876171, 4956824), (14.539598623409951, 2589532), (14.484065179475694, 4956825), (14.239105916662165, 8720406), (14.239105916662165, 3960437), (14.239105916662165, 2177460), (14.196073311981468, 8413644), (14.196073311981468, 6040011), (14.151533137296873, 6269131)]}, {'query_id': '1114993', 'scores': [(17.420279841642113, 4104358), (16.931023600474244, 8113957), (16.65461327441823, 6612208), (16.65461327441823, 6203023), (16.65461327441823, 6011208), (16.65461327441823, 4956693), (16.65461327441823, 3271410), (16.65461327441823, 2086636), (16.65461327441823, 1881796), (16.65461327441823, 1724571)]}, {'query_id': '1115210', 'scores': [(30.682138596822853, 8111887), (28.280927750114977, 8111878), (25.01774377894786, 8111885), (25.01774377894786, 8111884), (25.01774377894786, 8111882), (18.0045833542173, 8390254), (18.0045833542173, 7440361), (17.585872113421548, 7440364), (17.585872113421548, 7440363), (17.186193201752875, 7440366)]}, {'query_id': '1116380', 'scores': [(28.14200163383135, 8100548), (28.07687041053985, 8100546), (28.07687041053985, 8100543), (22.27332852011519, 521341), (19.70332907548651, 4118339), (19.70332907548651, 1422757), (19.70332907548651, 137869), (16.487795461508387, 5104772), (16.487795461508387, 1645711), (16.487795461508387, 444969)]}, {'query_id': '1117817', 'scores': [(24.499666748024932, 5310910), (24.499666748024932, 3516296), (23.052243389077656, 4486487), (22.002804946450745, 7531176), (19.95075308603358, 6537918), (19.809901772641894, 1005145), (19.38055600523617, 1035058), (19.225187363697685, 109686), (18.934715555604008, 5970406), (18.934715555604008, 5447234)]}, {'query_id': '1117886', 'scores': [(24.729390226048963, 2921720), (24.010935046127265, 556178), (23.834792168767095, 8456607), (22.876998560548586, 7460949), (22.876998560548586, 7460948), (22.876998560548586, 7024723), (22.876998560548586, 5376945), (22.787221349636653, 5094343), (22.6884832105242, 5301093), (22.204377160801492, 8407176)]}, {'query_id': '1118370', 'scores': [(15.470710112566096, 8656398), (15.253613147182286, 7476349), (15.253613147182286, 2951456), (15.107415753106865, 2224024), (15.089710843157427, 8081645), (15.089710843157427, 2224022), (14.872613877773615, 6211291), (14.872613877773615, 3608410), (14.872613877773615, 1284328), (14.650733424056131, 861060)]}, {'query_id': '1118426', 'scores': [(23.200704994973073, 8081194), (23.200704994973073, 8081193), (19.862302498626043, 8081199), (19.862302498626043, 8081196), (19.862302498626043, 2424347), (18.975664053272745, 6622100), (18.975664053272745, 5598458), (18.975664053272745, 2424353), (17.962655829523936, 6208983), (17.654318509251635, 2469356)]}, {'query_id': '1119118', 'scores': [(25.880494569290445, 8074735), (25.400849993589386, 8074733), (21.676724349900397, 6026130), (21.676724349900397, 5472178), (21.676724349900397, 3609036), (20.624716755145535, 8074741), (20.624716755145535, 8074734), (17.45168340820007, 5451461), (17.45168340820007, 5185340), (17.45168340820007, 4246892)]}, {'query_id': '1119543', 'scores': [(39.938129810117054, 8070706), (36.71542496427709, 8070713), (34.77740545218565, 1603603), (34.66043271334337, 405831), (34.21545751304812, 405832), (33.792449464323, 5052354), (33.67638909629049, 8070707), (31.893197993966883, 4770699), (31.546146272654095, 3612262), (27.51764225045892, 3612261)]}, {'query_id': '1120588', 'scores': [(27.748740126153486, 8060866), (27.350268133712483, 8060864), (26.83288152358641, 8060867), (26.23782631140057, 8060861), (25.66681007500109, 8060862), (25.51687160982233, 8060868), (24.156285305606275, 8060859), (23.78140105033075, 8060863), (23.78140105033075, 8060860), (23.21038481393127, 2313348)]}, {'query_id': '1121353', 'scores': [(31.50922075538937, 8053113), (30.788977921776272, 8053117), (26.584136568730628, 1854970), (24.74400697036654, 4872676), (24.153152654882824, 4210422), (24.153152654882824, 623601), (22.942093217390756, 6379159), (22.538424503581204, 5606064), (22.08151265489449, 5277905), (22.08151265489449, 937044)]}, {'query_id': '1121879', 'scores': [(34.53448664990921, 2603303), (34.527763517602764, 3024534), (33.80231069527568, 1307737), (33.62102190835542, 8048097), (33.62102190835542, 2603307), (33.62102190835542, 1406295), (31.217496305546106, 2193862), (31.154278788903063, 4995632), (31.10461838161462, 8048100), (30.385677869191586, 1025282)]}, {'query_id': '1122138', 'scores': [(15.26907931179721, 4342247), (15.224717169499295, 4342246), (15.197882472090502, 5871298), (14.661510841195334, 5463811), (14.50358063031867, 6283517), (14.359802270608938, 7067825), (14.359802270608938, 6963235), (14.090999250282046, 3967084), (14.090999250282046, 3647404), (14.090999250282046, 3345300)]}, {'query_id': '1122767', 'scores': [(30.243157098170446, 7544609), (30.243157098170446, 1575377), (29.51085732239147, 8039590), (29.51085732239147, 5877496), (28.735454187473536, 5877500), (27.651942506952526, 6851717), (27.651942506952526, 6179207), (27.651942506952526, 2859907), (27.280171233861253, 2019371), (26.23332940743189, 4550504)]}, {'query_id': '1122843', 'scores': [(42.28235712772559, 2332417), (42.26654329362261, 2332424), (41.473422277210375, 8772747), (41.24145930847786, 225969), (40.186842428067926, 7998359), (39.77492604532857, 4947308), (39.57683649216069, 2332416), (39.311821503388394, 3880592), (39.26386769953681, 2332425), (39.153128591318264, 3312892)]}, {'query_id': '1123657', 'scores': [(34.4669201973063, 8032346), (27.298421137384018, 8032342), (25.899726568369065, 8032349), (24.78104788750842, 7221025), (24.466361526784414, 5571459), (24.148603313839708, 8032348), (24.148603313839708, 8032347), (24.148603313839708, 8032343), (24.14075555668393, 7988749), (24.11868999377082, 5572500)]}, {'query_id': '1124552', 'scores': [(30.29621004688822, 5782491), (25.335584446195583, 3442729), (24.081040215422888, 4743535), (24.081040215422888, 3442725), (23.369063525289075, 7082651), (23.3022342623024, 8214271), (23.25592894279268, 2601066), (23.09182536281873, 2601070), (23.09182536281873, 1797406), (22.78502301765006, 6263843)]}, {'query_id': '1125632', 'scores': [(34.75759741403528, 7011199), (34.673996762661176, 6721396), (34.538599585805486, 1278069), (34.3998819242041, 7191566), (34.3998819242041, 1475286), (34.3998819242041, 1243662), (34.32533635566454, 6934400), (34.32533635566454, 6543904), (33.972843627735315, 7011204), (33.972843627735315, 5847127)]}, {'query_id': '1125755', 'scores': [(17.586188788934777, 1744931), (16.930797349218324, 7420642), (16.930797349218324, 2206347), (16.550873167654295, 7856574), (16.550873167654295, 3808440), (16.389609638094438, 7254544), (15.895481727937844, 7387332), (15.895481727937844, 7254547), (15.895481727937844, 2938068), (15.895481727937844, 482655)]}, {'query_id': '1126523', 'scores': [(23.72655829268987, 8805840), (23.07116685297342, 8805844), (23.07116685297342, 8805841), (20.988878489687192, 8805837), (18.04649914848064, 8805846), (18.04649914848064, 8556233), (18.04649914848064, 3214173), (18.04649914848064, 3025043), (18.04649914848064, 2841635), (15.96421078519441, 2774244)]}, {'query_id': '1126738', 'scores': [(40.10230637835696, 8790091), (40.10230637835696, 1091656), (39.295966924466946, 1091662), (39.295966924466946, 1091658), (36.08777408497411, 8790089), (36.08777408497411, 1091654), (35.32548892931486, 8790093), (35.32548892931486, 8790090), (35.32548892931486, 8790088), (34.20150059965, 8790087)]}, {'query_id': '1127004', 'scores': [(27.731932281356613, 6481277), (27.586924376055833, 6531608), (27.498631546733275, 695631), (27.497402001142227, 5224906), (27.353623641432492, 1756214), (27.297674131576997, 3961675), (27.229828526406383, 6011954), (27.153895771867266, 860288), (27.09457195340667, 6654716), (27.09457195340667, 4964550)]}, {'query_id': '1127233', 'scores': [(18.296917757692604, 8100081), (18.12342886544065, 8100077), (18.028882008816563, 8527129), (17.952526984327527, 5400207), (17.897534265338283, 6611628), (17.834510131956637, 124809), (17.785161046384218, 6438836), (17.785161046384218, 3842712), (17.785161046384218, 124808), (17.712858897754856, 6438837)]}, {'query_id': '1127540', 'scores': [(20.029711307171908, 8726963), (20.029711307171908, 8726962), (20.029711307171908, 186476), (19.11262462801214, 8726961), (19.11262462801214, 8726956), (19.11262462801214, 8726954), (17.866149108793955, 2574460), (17.866149108793955, 2044372), (17.866149108793955, 2044369), (17.866149108793955, 2044368)]}, {'query_id': '1128456', 'scores': [(36.6756963878081, 8649622), (31.52360483130378, 1459777), (27.698740155484423, 7526299), (26.940393668245, 4084616), (26.353199966557405, 8649615), (26.353199966557405, 3721820), (25.015360277560312, 3121928), (25.015360277560312, 3121927), (25.015360277560312, 2534794), (25.015360277560312, 2516294)]}, {'query_id': '1129081', 'scores': [(20.33838457106435, 8599986), (17.991647889787693, 7871513), (17.991647889787693, 6962228), (17.991647889787693, 6794362), (17.991647889787693, 4178203), (17.991647889787693, 3762080), (17.991647889787693, 2770798), (17.991647889787693, 2156392), (17.991647889787693, 1044198), (17.991647889787693, 687772)]}, {'query_id': '1130705', 'scores': [(16.243999726377528, 1468216), (16.16435490449481, 8442374), (16.16435490449481, 1621163), (16.108167119193993, 6762105), (16.108167119193993, 6762102), (16.03414194173418, 6628960), (16.03414194173418, 6233460), (16.03414194173418, 5379660), (16.03414194173418, 3570934), (15.93218273504425, 1910013)]}, {'query_id': '1130734', 'scores': [(34.952018860958475, 2546803), (34.08446194588046, 7212169), (33.509625508736946, 8460898), (33.446712141583916, 8460901), (33.25329924293893, 8460902), (30.666221222600228, 4311761), (30.666221222600228, 2810717), (30.666221222600228, 1081476), (30.408492042518155, 3823605), (29.862033728075932, 7427232)]}, {'query_id': '1130847', 'scores': [(36.87746220608539, 5474954), (35.13408642648099, 4201753), (34.131176649782695, 7332266), (34.131176649782695, 7332262), (33.684022437259614, 7332261), (33.59323396645793, 6209899), (33.43552579006536, 8299223), (33.08618191411966, 8299227), (33.03777405474772, 1458056), (32.40360029462333, 6149589)]}, {'query_id': '1131069', 'scores': [(30.09632102857106, 8432048), (29.600353982758953, 8432050), (28.455564586708732, 8191743), (28.455564586708732, 1780971), (28.455564586708732, 1780963), (24.928261880747293, 6907802), (24.62511391822229, 1780966), (24.157753746761543, 6809701), (24.157753746761543, 6809698), (23.953644009481273, 6809699)]}, {'query_id': '1132044', 'scores': [(28.64358307035746, 5858455), (28.13434697724154, 1014774), (27.762729668787674, 5858453), (27.71484125639293, 2121532), (26.354379477910225, 7645045), (26.19608102011078, 1691946), (25.328223732643163, 1818480), (25.236431753714413, 6166188), (25.15032003085051, 1650397), (25.10216726717845, 5810405)]}, {'query_id': '1132247', 'scores': [(39.440453116224546, 6246993), (37.55756787383823, 1210363), (37.55756787383823, 1210358), (37.55756787383823, 1210357), (36.429401632043366, 7390343), (35.698240637802556, 7390342), (35.698240637802556, 6525827), (35.697923428776654, 8238483), (35.26892143881293, 4623273), (34.90816340044835, 7390347)]}, {'query_id': '1132532', 'scores': [(32.61081477279093, 23966), (30.98069213940423, 6994907), (28.976786804282227, 4893092), (28.458069774306416, 6832793), (28.4390716070335, 8374347), (28.349024427076607, 3299150), (28.349024427076607, 2762578), (28.26730788979092, 5989422), (27.807227572517846, 1976995), (27.43803491711387, 6832795)]}, {'query_id': '1132842', 'scores': [(33.24534220672271, 3158781), (31.905914808862367, 7364932), (31.66055379503093, 834533), (31.31756706426125, 6536692), (31.263533374381886, 6570513), (31.06946968394593, 4874198), (31.006281745859162, 4132635), (30.88689219887928, 2324752), (30.77069406288725, 2941398), (29.304463280247305, 2391268)]}, {'query_id': '1132943', 'scores': [(26.466853964310864, 423711), (26.15561141626224, 5552077), (26.13011167012531, 6253953), (24.896488055394506, 622662), (24.736196475763812, 3437105), (22.212929025578777, 6469809), (21.979432678513913, 5782177), (21.908632296799873, 2107961), (21.904153352309205, 2248898), (21.904153352309205, 2107962)]}, {'query_id': '1132950', 'scores': [(22.216822423622848, 8274826), (21.584983949793042, 8619165), (21.292815493813052, 5867940), (20.67975815261899, 8274822), (20.67975815261899, 2303442), (20.45687669624945, 5845390), (20.44689964934932, 2132514), (20.44689964934932, 261777), (20.03470800062943, 6590922), (19.86792810588271, 6625752)]}, {'query_id': '1133485', 'scores': [(21.195613774906562, 5221748), (21.114944497121986, 8330714), (21.09727950941506, 3156351), (21.07712397359886, 3183897), (21.069207185748333, 2418248), (21.05401115419729, 6195711), (21.029694928400236, 8692792), (21.011153945298688, 5630061), (21.004189028734253, 4090344), (20.90125340724906, 3089617)]}, {'query_id': '1133579', 'scores': [(25.26114552648168, 8220878), (24.979719727590098, 4931644), (24.303968783094316, 1817055), (23.777824898210056, 4931642), (23.777824898210056, 4931641), (22.442228882734632, 670206), (21.84940301294373, 8220879), (21.84940301294373, 5335041), (21.84940301294373, 3691486), (21.84940301294373, 2448461)]}, {'query_id': '1134094', 'scores': [(23.896471655686263, 8172161), (22.5243037759774, 8172166), (22.5243037759774, 7182375), (21.789909141496715, 6426847), (21.789909141496715, 6426846), (21.789909141496715, 3445457), (21.104724157728796, 8172159), (19.92567660781497, 7182378), (19.92567660781497, 6426844), (19.92567660781497, 5357491)]}, {'query_id': '1134207', 'scores': [(15.228319401635972, 1698802), (15.191327174863668, 374550), (14.970134895081006, 7155676), (14.75899680983392, 5592724), (14.650529325877994, 8817805), (14.650529325877994, 853685), (14.622444896144334, 3225711), (14.601081406590058, 7049025), (14.4462067914618, 8246744), (14.364260389589369, 3362951)]}, {'query_id': '1134431', 'scores': [(32.93922278649441, 3128065), (32.20525197313406, 8141222), (31.6648153828383, 8141223), (30.926817660769352, 8141221), (30.889761566462973, 7974008), (29.794740508641226, 8141220), (27.57571512507313, 5010274), (27.498461198349204, 2339181), (27.43205066860595, 6720464), (27.43205066860595, 3243219)]}, {'query_id': '1134680', 'scores': [(44.4432504211804, 8118937), (38.3827845939378, 8118936), (37.10187389812646, 8118930), (31.602573354150778, 8796589), (31.402468424526234, 8307236), (31.186372891495466, 2456858), (31.186372891495466, 2200409), (31.186372891495466, 1865327), (31.186372891495466, 1865324), (31.186372891495466, 237951)]}, {'query_id': '1134939', 'scores': [(23.4130160178025, 322652), (23.39974024612587, 8093333), (22.62931932865123, 8093330), (20.018244021499164, 6204714), (20.018244021499164, 2090588), (18.131766708937956, 8455233), (18.131766708937956, 8455232), (18.131766708937956, 7543137), (17.71968110191664, 8093327), (17.71968110191664, 7543128)]}, {'query_id': '1134988', 'scores': [(43.88708698233529, 8088407), (42.73195457235513, 8088409), (40.66586177165798, 380755), (40.38742672882627, 8088404), (38.30234311034466, 380753), (38.30234311034466, 380752), (38.208489260602846, 7296294), (38.08360338046426, 380748), (37.72856232384763, 6432242), (37.1472107003645, 380749)]}, {'query_id': '1135268', 'scores': [(23.76332407435382, 8386270), (23.609666174644108, 2966472), (23.459880297612425, 5717144), (23.443873795950072, 6259848), (23.443873795950072, 6087160), (23.182308760782814, 4260), (23.15220180262491, 5225950), (23.15220180262491, 1683963), (23.0652654313396, 8371436), (22.968126755194888, 3622462)]}, {'query_id': '1135283', 'scores': [(19.160024054821314, 8061944), (17.86005783676075, 2582837), (17.464027571394123, 4502908), (17.227330631952874, 3368379), (17.12953803272207, 7770955), (17.12953803272207, 7043720), (17.12953803272207, 6979076), (17.12953803272207, 5959072), (17.12953803272207, 5834915), (17.12953803272207, 3989820)]}, {'query_id': '1135413', 'scores': [(37.89640488425962, 8050662), (36.352661311940224, 8050667), (32.74600576276093, 8050665), (32.57778997973306, 2115732), (31.771218646641586, 6672865), (31.603703236152995, 7522683), (31.402804761812433, 6039815), (31.402804761812433, 5287000), (31.402804761812433, 2061136), (31.349346448257897, 5118096)]}, {'query_id': '1135626', 'scores': [(28.431714333972817, 2003615), (28.25362000073719, 4035834), (28.234683243882827, 7676917), (28.2269101200263, 6532689), (28.075073105452102, 2902886), (27.53878171581926, 4619335), (27.036688180794464, 4892864), (26.684926846370335, 7714006), (26.642227741438248, 4079092), (26.516447282662917, 4035841)]}, {'query_id': '1136043', 'scores': [(26.52638863798932, 8728069), (26.487498712623086, 8728071), (25.44249113747719, 8728070), (25.434675022968655, 8728072), (24.902983900265028, 1056540), (24.208461581961334, 8728076), (24.150919448554262, 1056534), (23.77760495160242, 7018326), (23.77760495160242, 5326197), (23.46272150882153, 8728077)]}, {'query_id': '1136047', 'scores': [(32.326042468983964, 8724031), (31.027048056717845, 8724032), (30.26568087593763, 3988779), (29.782331669746572, 3597758), (29.782331669746572, 3006066), (29.126951142344993, 7009393), (28.50255000742722, 8328670), (28.361423901872143, 512482), (28.17470823960782, 2815435), (27.837113234677368, 8724036)]}, {'query_id': '1136769', 'scores': [(41.12814666903049, 8082043), (40.44931038635414, 189017), (39.45554432712372, 129247), (38.89953932790267, 189019), (38.27233292298898, 8082050), (38.267863572752134, 4713242), (37.74423966245326, 189016), (36.66437896352382, 8082046), (36.66437896352382, 5718075), (34.81155589631244, 5508127)]}, {'query_id': '1136962', 'scores': [(61.70544407762314, 7631079), (61.6448369503907, 6185710), (61.18298821151008, 6185711), (60.850036127471114, 6185717), (60.48410976984094, 8336574), (59.97488752073806, 6185714), (59.97488752073806, 6185712), (56.60620692951767, 596001), (54.77170492407001, 6185716), (54.50540094979132, 4601613)]}, {'query_id': '118440', 'scores': [(20.807841756611904, 7035722), (20.807841756611904, 1569786), (19.961749392044034, 8319567), (19.63093474397435, 4737856), (19.63093474397435, 1569787), (18.981466520683345, 8412702), (17.544938009402085, 6750552), (17.544938009402085, 6750548), (17.17252187241006, 6750547), (17.17252187241006, 2975214)]}, {'query_id': '119821', 'scores': [(23.018331525790302, 7093228), (21.827340546625507, 7172227), (21.827340546625507, 6527223), (21.05000934011354, 8012784), (21.05000934011354, 6267963), (21.05000934011354, 3365586), (21.05000934011354, 259724), (18.248940574181415, 8012780), (17.834191924768202, 2212944), (17.058792275865237, 2212940)]}, {'query_id': '121171', 'scores': [(21.7806784879984, 8260192), (18.85985278250794, 7165126), (18.85985278250794, 6930941), (18.85985278250794, 2337570), (16.16424828179059, 451171), (16.081212759795097, 8643275), (16.081212759795097, 451173), (15.82113923807334, 2550811), (15.82113923807334, 451172), (15.82113923807334, 259437)]}, {'query_id': '125659', 'scores': [(23.798640230794973, 6976559), (23.798640230794973, 4522348), (23.798640230794973, 1335471), (23.009272018113776, 829647), (22.840613873289826, 5881859), (22.529650637991175, 8290885), (22.529650637991175, 6635625), (22.529650637991175, 2745708), (22.529650637991175, 1106134), (21.740282425309978, 5693282)]}, {'query_id': '135802', 'scores': [(24.212925337728976, 8692623), (23.557533898012526, 8692626), (23.407148546706377, 8692628), (22.751757106989928, 8692619), (22.751757106989928, 6037338), (22.751757106989928, 6037331), (20.706323714394102, 8731214), (20.706323714394102, 7327478), (20.706323714394102, 5394718), (19.548365710972885, 8692624)]}, {'query_id': '141630', 'scores': [(40.65497717140021, 2473236), (38.86831859527273, 8319991), (38.770684055475016, 6290795), (37.88490925864993, 8319989), (37.64056963541792, 7985119), (37.510034379121095, 2366121), (37.38276492170364, 8319987), (37.294229413961865, 3500851), (37.111127493255644, 3500859), (36.91903383119926, 6496223)]}, {'query_id': '144862', 'scores': [(25.695940633888554, 92239), (25.600905731998598, 92237), (25.363787085846038, 92238), (25.29801130099314, 2865950), (25.045871686807246, 92233), (24.6143903306536, 5292762), (24.537855646036185, 6150954), (24.355228395776308, 8320087), (24.355228395776308, 7245936), (24.355228395776308, 92241)]}, {'query_id': '156498', 'scores': [(33.83615097903199, 4273410), (28.980756071320364, 5606985), (28.48960267328584, 8823842), (28.107039922246887, 3288004), (27.941711660497795, 5043014), (27.838842313294414, 7075146), (27.838842313294414, 3397582), (27.702680992441234, 3397578), (27.669779303055428, 3301559), (27.556165022331598, 3288005)]}, {'query_id': '166046', 'scores': [(30.983227422629792, 7947463), (30.983227422629792, 1076792), (30.983227422629792, 1076791), (30.242829285727453, 1076793), (28.14863777999792, 1076795), (28.14863777999792, 1076794), (25.32522428404817, 96920), (25.053632338445553, 7947466), (24.566443569476203, 96927), (24.37470165529678, 7865285)]}, {'query_id': '169208', 'scores': [(25.204019849539236, 4859433), (24.724813116269686, 4859432), (24.68243180040259, 4580137), (24.611275982230506, 4762140), (24.40556282083827, 7251639), (24.2867403489668, 4202275), (24.222755686792926, 1395417), (24.125985260433698, 85987), (23.99617407535326, 2888366), (23.992954817764154, 4762142)]}, {'query_id': '174463', 'scores': [(24.55258755738018, 8304456), (24.231330447206865, 4501665), (23.39947873976479, 102528), (21.767441165292652, 6294949), (21.675451423898956, 7089630), (21.675451423898956, 7089629), (21.22812176341202, 8304458), (21.053335409521388, 7402420), (20.003942266863135, 7611683), (20.003942266863135, 5793989)]}, {'query_id': '175920', 'scores': [(50.054641771875986, 8234235), (43.891150620497115, 101345), (39.158992979329824, 8537849), (39.06272661530688, 7615731), (39.06272661530688, 5673665), (38.58344432983391, 8537850), (38.19433162782845, 7360509), (38.19433162782845, 6815325), (38.19433162782845, 5717848), (38.19433162782845, 5673664)]}, {'query_id': '177604', 'scores': [(26.898029769775345, 8451995), (26.110126189805904, 8451992), (23.683750855733816, 5627757), (23.673914329138274, 2226834), (23.55288641976168, 5034647), (22.796060516442385, 3444466), (22.796060516442385, 1308162), (21.512377549550436, 8451989), (21.395914446677185, 8451987), (21.210926256417004, 8451991)]}, {'query_id': '181626', 'scores': [(22.17998269427472, 43641), (21.789374683128194, 436351), (21.572277717744385, 7795335), (21.572277717744385, 4798089), (21.572277717744385, 2183331), (21.41179343214739, 71074), (21.175229164501417, 8069297), (21.05910753185426, 71073), (21.021185421000865, 5427354), (21.021185421000865, 4091013)]}, {'query_id': '197312', 'scores': [(22.510948286643774, 2317108), (22.510948286643774, 2317106), (22.26049698047209, 5823013), (21.88346622348719, 5823008), (21.84886763726748, 1179432), (21.649153247366392, 5823015), (21.649153247366392, 1179433), (21.62064766680036, 1665689), (21.508506056264558, 2317100), (21.508506056264558, 1239571)]}, {'query_id': '206106', 'scores': [(32.29102125814695, 6479522), (31.509471106558244, 75400), (31.192603399299177, 7079997), (30.88476257357937, 1390597), (30.73572437953029, 5093731), (30.539732884620754, 8427974), (30.40522823746061, 8737574), (30.299612587932707, 3217609), (30.299612587932707, 2947646), (30.069423137114338, 7118029)]}, {'query_id': '227873', 'scores': [(31.899683996185864, 7276796), (31.17838055137169, 8453473), (30.56089975792402, 7276798), (30.56089975792402, 3250369), (30.56089975792402, 2054488), (28.766002403432232, 7019128), (28.406332052498705, 3324178), (25.994068261564976, 1900788), (24.621556462492308, 8453475), (24.50177155842976, 8453476)]}, {'query_id': '23849', 'scores': [(22.860597789108628, 4834498), (22.27352735354031, 2674124), (22.173121673120644, 6728787), (22.126879823743685, 4348282), (22.063188219884857, 5888570), (21.820002096060247, 2373852), (21.562268941217155, 641196), (21.539809388175367, 8246990), (21.539809388175367, 2017213), (21.40986282334399, 5363400)]}, {'query_id': '240158', 'scores': [(31.741475719500567, 2043788), (30.843099851886763, 991184), (30.830440420173588, 6536692), (30.485525404931302, 1567595), (30.252015740801763, 4874198), (27.89123712554475, 2819781), (27.84293556204974, 2819779), (27.410288428741403, 3542898), (27.11271830375189, 2819784), (27.0799611296849, 6239283)]}, {'query_id': '245052', 'scores': [(29.981277797819015, 3828386), (26.70745007120629, 5023796), (26.281694150742826, 4205737), (26.281694150742826, 66616), (26.251352001737093, 5878365), (26.032122410503554, 5666135), (26.01914731175632, 8282910), (25.933530914990836, 5143412), (25.812627073826402, 8279354), (25.812627073826402, 3379257)]}, {'query_id': '246883', 'scores': [(31.947165065529035, 3050794), (31.46147942349078, 8699192), (30.549467905360657, 5050393), (29.304231348419425, 8699195), (28.709834806392166, 4959448), (28.367187711360636, 5683913), (28.367187711360636, 2413876), (28.138464770189216, 6242496), (27.901853056598764, 8420183), (27.896254828566214, 6083923)]}, {'query_id': '253749', 'scores': [(24.719735608276025, 6190136), (24.186079731167865, 3516374), (23.944970631298283, 4070966), (22.252642351081835, 3516370), (21.92961528859971, 3516368), (21.064756127693425, 8254833), (20.254665864618588, 8254894), (19.56705236583631, 8521273), (19.51057179960666, 6260689), (19.21936747620524, 6447304)]}, {'query_id': '256942', 'scores': [(33.64069705355164, 2602557), (28.428883286631283, 2364119), (28.000785367134856, 7213042), (27.856222778788183, 4890686), (27.6743279167385, 8327014), (27.614299770651535, 3262199), (27.491327689708044, 5297913), (27.264860564033487, 5083782), (27.264860564033487, 4604331), (27.264860564033487, 1183278)]}, {'query_id': '257119', 'scores': [(30.51685014383154, 8499023), (29.86652093149418, 4363515), (27.52812150069832, 4363509), (26.007957264748203, 8032136), (25.444009205490442, 3818024), (25.372574632247918, 889398), (25.165902933586842, 8276379), (25.07003459259692, 2112755), (25.062662236984956, 5853762), (25.040978672178568, 4212490)]}, {'query_id': '258062', 'scores': [(36.51788452841922, 8694161), (36.27712417935941, 4159145), (35.63537694391206, 18103), (34.672052052926176, 3683402), (33.82109682576717, 7918982), (33.46122820737522, 2707010), (33.43838746068586, 4174313), (33.01889781798552, 3195767), (32.6221264879278, 91493), (32.41766764840526, 4174307)]}, {'query_id': '26703', 'scores': [(20.395792430012904, 3532417), (19.417386200163985, 3532415), (19.417386200163985, 1485077), (16.537761855156525, 380356), (16.003728516849062, 7265225), (16.003728516849062, 1236259), (15.97302594746483, 3733548), (15.665774569613554, 3921274), (15.618295759635853, 3132961), (15.618295759635853, 524170)]}, {'query_id': '273695', 'scores': [(31.416475771166045, 8282608), (30.81476854208236, 5542707), (30.329022199140624, 7381998), (30.270252777816594, 5542705), (29.981755214097827, 2714742), (29.699236541417115, 7381994), (29.37922875212662, 5719948), (29.37922875212662, 4171983), (29.37922875212662, 413461), (29.351969556374318, 413460)]}, {'query_id': '302846', 'scores': [(36.60916059130037, 8418433), (35.450793988797166, 8418435), (33.60031560537268, 447099), (33.362508274114205, 8283558), (33.292011336382025, 6263014), (33.21030622368226, 3892531), (33.18808682468977, 3566357), (33.079640512876765, 5696466), (32.92197194703096, 447101), (32.757052392792374, 4234633)]}, {'query_id': '318362', 'scores': [(29.383983202945302, 5654804), (28.72985848763762, 429301), (28.604555027130672, 3145979), (27.87838537775572, 4299889), (27.445150297579758, 1714531), (26.977015274214843, 1714527), (25.843795764722884, 4299887), (25.68945192903773, 543937), (25.16392818442806, 519111), (24.903429412964027, 3996891)]}, {'query_id': '324585', 'scores': [(31.039755853781863, 1051441), (28.47199444235713, 8290314), (27.261079373026295, 8290310), (26.780027247391764, 3608325), (26.142297466470826, 7977807), (25.74235121954372, 7941125), (24.729877229439474, 7359510), (24.510437749852315, 8310496), (24.459079176254214, 405282), (24.394097335190505, 6393239)]}, {'query_id': '330501', 'scores': [(28.958924433613813, 6006326), (28.44830037274712, 3718639), (28.44830037274712, 1750071), (26.427507273524906, 6647777), (26.31067911457542, 7354312), (26.31067911457542, 4397700), (26.31067911457542, 452513), (26.152101866227834, 1899109), (26.152101866227834, 294036), (25.61367643802586, 2169021)]}, {'query_id': '330975', 'scores': [(41.088286607433396, 7908218), (37.610280588215254, 5750875), (37.53937716809098, 233948), (37.49374489168959, 4108178), (37.29194981713199, 8327008), (37.263074404068774, 3297858), (37.25454564292699, 233945), (37.065790633347206, 3297857), (36.87522912505494, 1097861), (36.8429839387353, 8541461)]}, {'query_id': '332593', 'scores': [(39.211272538763744, 8434873), (38.73258522332392, 5241323), (37.93113869646589, 5241322), (37.12050580237126, 7073995), (36.49184923831097, 4431502), (36.49184923831097, 1650687), (36.14482099967, 8434869), (36.14482099967, 7073996), (35.56261912674753, 8434872), (35.11561756811462, 4614346)]}, {'query_id': '336901', 'scores': [(32.94450233763519, 3646550), (32.236238437667744, 2909600), (31.809790950654445, 8528674), (31.62490815321239, 8130980), (31.26004644535117, 7836261), (31.012519561890457, 8528681), (31.012519561890457, 8528680), (31.012519561890457, 8528679), (31.012519561890457, 8528678), (31.012519561890457, 8528677)]}, {'query_id': '3505', 'scores': [(22.010006573343226, 8285655), (21.950573672823705, 8285658), (21.821269440561743, 4520880), (21.803975274187355, 5469617), (21.724134375737677, 8285654), (21.724134375737677, 8285653), (21.715678058671784, 8285656), (21.671679253601067, 502056), (21.6687362441714, 1030851), (21.613712573904724, 1076196)]}, {'query_id': '360721', 'scores': [(29.756893699578637, 6318492), (27.003000190924112, 4284999), (26.993029713640176, 101662), (26.604899540437344, 7043084), (26.187873681420328, 7826496), (26.183977570882412, 101666), (26.02447904676214, 7826493), (25.803764067936655, 1750204), (25.575191837905404, 7826491), (25.502795635909095, 6206175)]}, {'query_id': '384356', 'scores': [(32.796880843382965, 7565188), (25.666467207721283, 2032126), (25.42274513790461, 2032127), (25.17235320773267, 3705729), (25.115843325480615, 5381456), (25.09716880488639, 6863040), (25.08750686610932, 4648730), (25.08750686610932, 412129), (25.069572621495205, 6543899), (25.069572621495205, 2032125)]}, {'query_id': '390360', 'scores': [(29.20812774772065, 604111), (27.170007397635107, 5423359), (26.787263841300792, 2203877), (26.787263841300792, 2203875), (26.44333262498268, 7679050), (25.962586354064108, 6179316), (25.366885925761537, 4683960), (25.366885925761537, 324991), (25.149788960377727, 324994), (25.149788960377727, 324992)]}, {'query_id': '405163', 'scores': [(21.633787859669223, 851536), (20.9963558838136, 6289824), (20.47542125716602, 7622492), (20.29594817067213, 429247), (20.29594817067213, 429246), (20.29594817067213, 429245), (19.137581568168926, 3293026), (19.137581568168926, 2748170), (19.137581568168926, 1518541), (19.137581568168926, 1352548)]}, {'query_id': '42255', 'scores': [(44.796906050631826, 8746197), (42.03389089148414, 8746189), (39.58768618438535, 8746188), (38.51257724374411, 8746194), (36.98244931879543, 3723425), (36.95837469015282, 3063186), (36.76714457413191, 2882385), (36.69649631230102, 7123389), (36.66907950442223, 8746196), (36.658266924874205, 6784918)]}, {'query_id': '425632', 'scores': [(30.05978476807752, 8454642), (30.05978476807752, 8454639), (28.948857275189894, 8454636), (27.69793402030919, 8454635), (27.69793402030919, 650345), (21.399799199476647, 8454641), (18.107522399557162, 8454643), (18.107522399557162, 8454640), (18.107522399557162, 8292459), (18.107522399557162, 6843069)]}, {'query_id': '426175', 'scores': [(19.99698658838695, 330964), (19.563188175308206, 6209130), (19.563188175308206, 4362255), (19.440465475504972, 6758532), (19.440465475504972, 5030738), (18.971887894438083, 6049564), (18.971887894438083, 3017346), (18.504364527577383, 6016624), (18.432005029295798, 3870458), (18.432005029295798, 3562170)]}, {'query_id': '42752', 'scores': [(21.423736085444553, 4303156), (21.377693905347037, 1254236), (21.351903583919572, 8660833), (21.351903583919572, 5932129), (21.346392079666288, 7920885), (21.1993968207006, 5325520), (21.146659474456193, 3818076), (21.13845897834657, 4821761), (21.009457638017583, 4747962), (20.99866349380827, 1986201)]}, {'query_id': '435548', 'scores': [(18.71216751526859, 8547993), (18.71216751526859, 8547992), (18.687453168331672, 7875423), (18.38619194287303, 7875421), (18.10572205791035, 7080768), (18.10572205791035, 235984), (18.075033441949994, 7875428), (18.075033441949994, 6064445), (18.075033441949994, 1394884), (17.49330233152867, 6424374)]}, {'query_id': '436707', 'scores': [(21.565781385907616, 1595287), (21.20211811043217, 2203550), (21.20211811043217, 2101427), (21.20211811043217, 1825031), (20.90723461007642, 5413961), (20.90723461007642, 3370495), (20.467506137719894, 5044105), (20.242726122362434, 8641216), (20.242726122362434, 8200621), (20.135352753267497, 6045618)]}, {'query_id': '444389', 'scores': [(23.45819939258872, 5896071), (22.97346869566392, 8495210), (19.549071666861824, 8044142), (19.549071666861824, 7705876), (19.549071666861824, 5208202), (19.272489486756484, 1722558), (19.122907863739485, 7484162), (19.122907863739485, 7150173), (19.122907863739485, 1048029), (18.88267071162128, 5251228)]}, {'query_id': '449367', 'scores': [(24.543231460340586, 8432230), (24.480981511830166, 635517), (24.166167515625446, 6632652), (24.026111679597417, 6632645), (24.026111679597417, 6632644), (23.616936435724867, 8600273), (23.11454875078192, 5717147), (22.967254871533033, 2596719), (22.941431198449777, 386955), (22.807210077816293, 1998108)]}, {'query_id': '452915', 'scores': [(26.47763572856643, 135840), (25.66301706956659, 2447912), (25.659457919932255, 3332642), (25.471092251775115, 1044828), (25.308047454635002, 953667), (25.202289231448223, 1364570), (24.98866382648245, 8366603), (24.98866382648245, 8366600), (24.98866382648245, 3779965), (24.9812473607547, 7261641)]}, {'query_id': '463271', 'scores': [(29.096906825772702, 6239584), (28.605932870375042, 8604216), (28.605932870375042, 8326637), (28.605932870375042, 6239587), (28.605932870375042, 3914186), (28.13407918316062, 2455683), (27.962043990582742, 8326635), (27.874478361005238, 8435508), (27.874478361005238, 3914192), (27.690099562680878, 6005616)]}, {'query_id': '469589', 'scores': [(34.36907003947154, 8581474), (32.451013250924284, 8581470), (32.451013250924284, 8581468), (30.51104835490213, 2210168), (28.98606515801778, 4871853), (28.65456929237459, 2210175), (27.148862851847483, 8581475), (27.148862851847483, 8581466), (27.148862851847483, 2210172), (26.786583171555897, 6441572)]}, {'query_id': '47210', 'scores': [(39.40735838251883, 2042918), (38.92777190973074, 6180504), (38.536591402233675, 2338385), (37.31869673715809, 5207329), (37.306902287019604, 2545998), (36.625264575683616, 4748997), (36.017626027731254, 3858013), (35.860807168286804, 2546002), (35.860807168286804, 2546001), (35.8316351751634, 7367873)]}, {'query_id': '482726', 'scores': [(13.859663652421483, 8730812), (13.66661662883713, 8572751), (13.377394479799944, 6318274), (13.377394479799944, 4734241), (13.377394479799944, 22062), (13.367067288601746, 1272524), (13.285759535328626, 4734239), (13.214007644846369, 4768911), (13.214007644846369, 3512937), (13.214007644846369, 1985718)]}, {'query_id': '48792', 'scores': [(32.86343825946438, 64048), (32.54030528565261, 64047), (29.789532277819003, 64041), (20.073607219357562, 6619302), (19.72312698412783, 5803986), (19.72312698412783, 3820076), (19.541548782241556, 3820082), (19.5062156411961, 6152219), (19.504387254247433, 6152218), (19.504387254247433, 3820080)]}, {'query_id': '50122', 'scores': [(28.678499136309696, 8273511), (27.08316087286957, 1417366), (26.685286289400587, 8273514), (26.437411560413253, 8273517), (26.437411560413253, 425197), (24.798201767254916, 7341000), (21.40532983806166, 8361668), (21.270651878017254, 8337928), (21.270651878017254, 5999140), (20.91664882426648, 8273519)]}, {'query_id': '514096', 'scores': [(14.269459064819728, 4786295), (14.120858133037931, 6286608), (14.120153497916533, 6474925), (14.120153497916533, 6285508), (14.040509954807074, 1837337), (14.004171081505467, 3534826), (13.92382290327461, 8460993), (13.92382290327461, 5936453), (13.911465910981759, 7016672), (13.911465910981759, 3708394)]}, {'query_id': '519025', 'scores': [(18.300780777498236, 4232478), (18.233437569483172, 288148), (17.898975285501816, 2692126), (17.80945291058821, 6805536), (17.80945291058821, 6526749), (17.665674550878474, 6526750), (17.665674550878474, 3394216), (17.665674550878474, 1697202), (17.649222141983184, 6236703), (17.649222141983184, 5931056)]}, {'query_id': '53233', 'scores': [(31.724497861247475, 8333467), (30.782396091072492, 8257054), (30.57942593792626, 8333470), (30.57942593792626, 8333463), (30.19995999943549, 8257048), (30.19995999943549, 5256809), (29.23670030942045, 8257051), (29.23670030942045, 7355044), (29.23670030942045, 5516316), (29.23670030942045, 579087)]}, {'query_id': '537060', 'scores': [(25.412451467483137, 8420027), (25.375496254269283, 8420024), (25.012797561641428, 8420032), (25.012797561641428, 8420025), (23.998291492966317, 8420030), (23.998291492966317, 8420029), (23.998291492966317, 8420026), (23.998291492966317, 7386519), (23.998291492966317, 1086305), (23.1412116039733, 8420031)]}, {'query_id': '537817', 'scores': [(28.071834693812697, 1956211), (27.99187457017451, 6665873), (27.937850821796612, 8527917), (27.837353060738288, 540370), (27.823947882452188, 5264609), (27.746871090648984, 1956207), (27.376680640698915, 3759425), (27.353119364751954, 540368), (27.327665861049645, 2025709), (27.222159131262693, 1095538)]}, {'query_id': '543273', 'scores': [(30.8387550403827, 8507214), (30.16653291555845, 8507211), (28.698847922063155, 8507209), (28.456521921370953, 4308075), (27.78928550256225, 5990522), (27.211308317433822, 8507212), (26.895980091042137, 8507208), (26.895980091042137, 4308073), (26.895980091042137, 3143117), (26.095531516355546, 4067764)]}, {'query_id': '545355', 'scores': [(35.63022198335884, 4290924), (34.50511598461808, 4290920), (33.790504015447745, 8547218), (33.790504015447745, 8547215), (33.62184008040103, 8547219), (33.62184008040103, 4290921), (33.306511854009344, 4290922), (30.263901368617933, 4290923), (26.369290873392224, 4290918), (26.369290873392224, 3294315)]}, {'query_id': '555530', 'scores': [(27.379980506977045, 1897716), (26.89954792132373, 7475084), (26.834614268454494, 4296807), (26.482235115237067, 4137963), (26.40644589630039, 1754575), (26.380465719765265, 6485976), (26.066555328310393, 6150835), (26.058963918578577, 3948935), (25.836781873969752, 4238674), (25.674586359413343, 1483513)]}, {'query_id': '583468', 'scores': [(20.353395278900887, 7491704), (20.250393351859575, 8326078), (20.170026681747185, 7491700), (20.170026681747185, 1675562), (20.09663471522896, 7706119), (20.09663471522896, 2267054), (20.09663471522896, 2267052), (20.09663471522896, 1675564), (20.005090260541806, 6654310), (20.005090260541806, 2080306)]}, {'query_id': '586148', 'scores': [(23.233863939573926, 5056079), (22.515785037124484, 960743), (22.45176908654177, 8271038), (22.405142397873863, 6148926), (21.985929369464728, 7779366), (21.898998477293198, 6854043), (21.898998477293198, 1658943), (21.875286730214103, 2811037), (21.848540970336344, 8567688), (21.816326381767414, 5678812)]}, {'query_id': '590019', 'scores': [(22.028152614086693, 4408763), (21.960541219746414, 2831848), (21.957367279684515, 2831847), (21.925466825791865, 632727), (21.6468705147638, 5075460), (21.497372467317575, 3453506), (21.468076822950728, 3266922), (21.440115200181243, 1248322), (21.438139494170564, 6066327), (21.430754974647755, 8369826)]}, {'query_id': '605127', 'scores': [(29.54948404811826, 1137979), (29.29054441109448, 8642036), (28.217137581717346, 8594907), (28.217137581717346, 1137978), (27.90405594351246, 5121398), (26.968448798792522, 3679424), (26.531213568429052, 8547782), (26.270135721135908, 8642038), (26.270135721135908, 1137982), (25.48261988704425, 6888324)]}, {'query_id': '610265', 'scores': [(31.78958420887561, 474853), (31.477248824789537, 222769), (31.46975308639441, 2877322), (31.46975308639441, 683818), (31.362871495751147, 2877320), (31.35522339237378, 1148168), (31.30033911772799, 4758657), (31.29245936379738, 1148169), (31.2644875539485, 7009719), (31.23099309757871, 2039693)]}, {'query_id': '611953', 'scores': [(41.95952831774613, 484756), (36.60579472237875, 7197690), (33.34764124114653, 3562872), (33.34764124114653, 484758), (32.84694580888977, 456354), (31.697129061277842, 8062950), (31.67903804615015, 456353), (31.40394501555815, 456359), (31.313295934986996, 8062949), (30.86999404774277, 8371935)]}, {'query_id': '640502', 'scores': [(24.137307504544477, 3551380), (23.979533107070864, 5827712), (23.7259897108563, 5827713), (23.245176778833656, 6730887), (23.245176778833656, 3271456), (23.245176778833656, 1861075), (23.24228895824674, 6985038), (23.218890447598476, 5150987), (23.127463852892276, 3678130), (23.071816021131035, 77134)]}, {'query_id': '64647', 'scores': [(27.09904634719087, 8293649), (26.778668501629156, 8293652), (25.15409869298266, 4161653), (23.884000404013776, 8293648), (20.012602224013186, 8599633), (19.33726995474058, 1422031), (19.31166950715128, 6508277), (19.12412981749385, 7509908), (19.12412981749385, 5692182), (19.12412981749385, 1310940)]}, {'query_id': '653399', 'scores': [(23.014136012978, 1969533), (21.896634527665178, 7133996), (21.532149073346694, 8458007), (21.436484406288333, 7485466), (21.436484406288333, 6343750), (21.436484406288333, 3111946), (21.436484406288333, 2007733), (21.12100887379972, 1674561), (20.9902792149612, 2399252), (20.965139502650423, 7092040)]}, {'query_id': '655526', 'scores': [(24.431139065384585, 6040076), (23.742330237490634, 8425101), (22.977834387595756, 206999), (21.00283059470325, 6707899), (21.00283059470325, 3864910), (21.00283059470325, 3021665), (21.00283059470325, 2105608), (18.570004216190537, 1826238), (18.310887878290202, 5883282), (17.894731335601787, 7135528)]}, {'query_id': '655914', 'scores': [(19.661471306404803, 8613490), (19.067925850907507, 1270195), (17.56123354079898, 8613495), (17.56123354079898, 8058173), (17.56123354079898, 5866957), (17.56123354079898, 5235329), (17.56123354079898, 2789189), (14.473927734054586, 3209412), (14.399575365557732, 6248270), (14.399575365557732, 2519335)]}, {'query_id': '660198', 'scores': [(19.106625083618113, 2845619), (18.248321816491032, 523901), (18.228771270922223, 2410610), (18.08250027555718, 4434042), (18.032999183188267, 7327517), (18.032999183188267, 4264632), (18.032999183188267, 3556727), (17.784910062208994, 7327515), (17.784910062208994, 4264630), (17.784910062208994, 3604202)]}, {'query_id': '67316', 'scores': [(37.90866063178886, 6680964), (33.769282667720894, 7630885), (33.20283994442197, 6764671), (32.52083576535737, 6179107), (32.397210962163044, 4163705), (32.129180411721094, 7112302), (32.10955148724639, 4252546), (32.10955148724639, 3810719), (31.581609540089765, 8431637), (31.458095581147816, 1974375)]}, {'query_id': '673670', 'scores': [(17.282007517506695, 3607501), (17.118969710737762, 7257303), (17.118969710737762, 2240359), (16.880100365936773, 8008345), (16.880100365936773, 3607497), (16.880100365936773, 1905068), (16.880100365936773, 1398756), (16.496461721256388, 2240358), (16.496461721256388, 2240357), (16.496461721256388, 2240355)]}, {'query_id': '701453', 'scores': [(24.206111022723093, 4355647), (24.206111022723093, 4330679), (23.84573194781223, 4355649), (23.754391721719188, 8626149), (23.567346797193544, 6999087), (23.457943162093635, 4330677), (23.233932527084256, 2036081), (23.186607907625962, 4355645), (23.126317639001012, 2036076), (23.046887602558613, 4330673)]}, {'query_id': '703782', 'scores': [(21.86505116274649, 4061595), (21.488922667992632, 4061598), (21.05555722838493, 4061597), (21.05555722838493, 344218), (21.05276311422473, 185599), (20.759962887052026, 4647484), (20.70567598019244, 4948640), (20.634155134647266, 3490915), (20.634155134647266, 611402), (20.53564529078316, 7687244)]}, {'query_id': '708979', 'scores': [(20.3801296232292, 6157147), (20.340027978512836, 4805197), (20.293813233439856, 279265), (20.254024012882986, 3811546), (20.23493691437428, 2889574), (20.1705682596444, 6313188), (20.157369700367248, 1447346), (20.134207230448848, 8088373), (20.11169194057883, 6409763), (20.11169194057883, 3422810)]}, {'query_id': '716113', 'scores': [(22.727449504139592, 8520612), (22.727449504139592, 8520611), (22.41032230175625, 8520617), (20.94877954294606, 8520618), (20.94877954294606, 8520614), (20.94877954294606, 8520613), (18.53161267260613, 8520616), (18.53161267260613, 8520610)]}, {'query_id': '730539', 'scores': [(28.021698298220315, 8577877), (26.802454475047874, 8276218), (26.368731031343962, 2666444), (26.117734758151407, 2666436), (25.328818562596783, 8577880), (23.326185143111964, 8577876), (23.326185143111964, 5428552), (23.326185143111964, 4605022), (18.145072497756377, 8276217), (18.145072497756377, 7916064)]}, {'query_id': '735482', 'scores': [(20.95904057147796, 8405801), (20.704858530079424, 8405802), (20.419876258588182, 4823368), (20.241858271284883, 8405807), (20.241858271284883, 338960), (20.02617748777707, 8405806), (19.57256803445122, 8405805), (18.465664950457082, 4117288), (18.421097883700984, 8405804), (18.421097883700984, 8405803)]}, {'query_id': '735922', 'scores': [(20.902884347726427, 5537112), (20.902884347726427, 5537109), (20.129240188307204, 5854095), (20.129240188307204, 281701), (20.129240188307204, 281697), (20.129240188307204, 180247), (19.569426147540504, 8626887), (19.264657236254145, 7371194), (19.264657236254145, 5861444), (19.264657236254145, 1432224)]}, {'query_id': '75198', 'scores': [(24.390280548779728, 7797928), (24.181549528186487, 2579716), (23.778621801776985, 8256960), (22.848041523603513, 6862216), (22.318185855943753, 6862217), (22.318185855943753, 3916717), (22.318185855943753, 2140526), (22.318185855943753, 2140522), (21.04990302096848, 7088445), (21.04990302096848, 2932859)]}, {'query_id': '768208', 'scores': [(20.624078671734345, 52162), (20.33630082980317, 52155), (19.874112174580368, 8525681), (19.874112174580368, 52159), (19.010020340902962, 3515884), (19.010020340902962, 3515882), (19.010020340902962, 813102), (19.010020340902962, 52160), (19.010020340902962, 52158), (19.010020340902962, 52157)]}, {'query_id': '779302', 'scores': [(26.110665801292683, 1583173), (25.545699157192303, 8768560), (25.545699157192303, 8768554), (24.111553060629795, 8768563), (24.111553060629795, 8768562), (24.111553060629795, 8768561), (24.111553060629795, 8768558), (24.111553060629795, 8768555), (20.048992815996883, 2411546), (19.742264120271315, 4638389)]}, {'query_id': '792635', 'scores': [(21.020247038837134, 4294317), (20.10632325453987, 8682263), (20.10632325453987, 8682259), (20.10632325453987, 8682258), (20.10632325453987, 8682254), (20.10632325453987, 6306494), (20.10632325453987, 6306491), (20.10632325453987, 4294318), (20.10632325453987, 4294312), (20.10632325453987, 4294311)]}, {'query_id': '794223', 'scores': [(19.43874145001271, 7790213), (19.167779422391188, 2519053), (18.42471838424802, 8445687), (17.234891954328624, 471681), (17.195809744242013, 6209396), (17.073661729071144, 5946388), (17.073661729071144, 892194), (16.330600690927977, 843980), (16.330600690927977, 843973), (16.330600690927977, 520630)]}, {'query_id': '794429', 'scores': [(25.42923530959834, 1208450), (24.305882164771873, 1208449), (24.13566172070159, 64014), (23.009588013775137, 8618468), (23.009588013775137, 4652685), (23.009588013775137, 719934), (19.820298316040052, 6098420), (19.816250855773056, 1246185), (19.7502210958559, 3926405), (19.6078896007601, 4627633)]}, {'query_id': '801118', 'scores': [(25.97279122372523, 2932991), (25.487500501928423, 4799238), (25.38640740241662, 7615528), (25.38640740241662, 3975963), (24.59157977764139, 1930307), (24.503542617041948, 3263703), (24.147630496117202, 7486725), (23.90534586247828, 4214955), (23.873215102259547, 3360132), (23.86042062132546, 5219693)]}, {'query_id': '804066', 'scores': [(15.725771106763728, 73387), (15.555198781648528, 6152070), (14.729165959101302, 8567803), (14.729165959101302, 7922526), (14.729165959101302, 7621407), (14.729165959101302, 5655169), (14.729165959101302, 1504069), (14.729165959101302, 1259677), (14.363198580400871, 7427197), (14.363198580400871, 3298496)]}, {'query_id': '808400', 'scores': [(22.135317215125845, 3691917), (21.36585660255728, 3691923), (20.756014685824034, 3377142), (19.927255157955777, 7282136), (19.927255157955777, 7252539), (19.927255157955777, 4576228), (19.927255157955777, 1995351), (18.53094378838643, 5834423), (18.421902682259145, 5598346), (18.109785975014013, 5834421)]}, {'query_id': '809525', 'scores': [(27.42538122228916, 3793929), (24.573920821126393, 2077416), (24.020395911137925, 1724922), (23.50350983560186, 1428570), (23.20743837211966, 3793922), (23.20743837211966, 1428575), (22.653913462131193, 7059360), (22.653913462131193, 1428574), (22.653913462131193, 1428573), (22.615294216551817, 2077420)]}, {'query_id': '814183', 'scores': [(31.153893624466157, 8679359), (24.38831177251556, 8444060), (22.60609281146659, 8679357), (22.03700036792222, 7274881), (20.12821248945201, 510865), (20.12821248945201, 510863), (19.67075311469174, 510861), (18.81550297927036, 7013314), (18.81550297927036, 878335), (18.81550297927036, 510868)]}, {'query_id': '819983', 'scores': [(15.629884651305396, 8145644), (15.629884651305396, 1338142), (15.57585673406358, 1617043), (15.539899738217027, 4580581), (15.538363451105887, 8312879), (15.536364709319958, 8328822), (15.507709766537394, 8439123), (15.501440923522967, 8831812), (15.501440923522967, 8045345), (15.50007042902299, 1221184)]}, {'query_id': '849550', 'scores': [(22.22713073927145, 1373164), (22.165456766370884, 7274031), (21.96997360322762, 6758594), (21.60791069780627, 8582102), (21.557392223190995, 7767168), (21.557392223190995, 4061418), (21.339107677479376, 8000555), (21.339107677479376, 7767169), (21.339107677479376, 1419091), (21.330688592998968, 5690724)]}, {'query_id': '85020', 'scores': [(31.134208105931577, 3152840), (30.938365088366425, 198481), (30.649419274089674, 198476), (30.42571287876232, 6290170), (30.08813157514495, 6974021), (29.939957010168982, 1770218), (29.20986148918064, 603541), (29.14034554155876, 147291), (29.028698716590277, 1640375), (29.001321691242758, 1777604)]}, {'query_id': '850358', 'scores': [(29.165753578345395, 2852825), (29.165753578345395, 2852824), (28.56096101171795, 2852830), (28.00875910305811, 6427506), (27.478041568035604, 2852827), (27.442725295355626, 2852826), (24.776979206551403, 8037013), (22.871289688750835, 389081), (22.835507535477, 2278756), (22.832537208035, 6427505)]}, {'query_id': '86606', 'scores': [(27.076564631775533, 5013458), (26.989033078409605, 8422274), (26.977449490254234, 6253490), (26.977449490254234, 6253489), (26.977449490254234, 677277), (26.861574887309366, 3077883), (26.76826153125031, 747160), (26.42624971078849, 6897895), (26.32043388675848, 677274), (26.32043388675848, 677272)]}, {'query_id': '877809', 'scores': [(30.542379035335212, 4888926), (30.33254311591184, 5940394), (30.33254311591184, 5940393), (30.10862990554012, 6704838), (29.9203524567138, 741714), (29.1166392035293, 4888929), (28.97115142078609, 7040022), (28.963444473452608, 7040017), (28.930853284762634, 1550610), (28.66188495571084, 7346771)]}, {'query_id': '883915', 'scores': [(32.02175426985656, 4042980), (31.238254073296368, 6172403), (30.849592862342558, 4042985), (30.803892593077062, 3535079), (30.644981344791262, 5406348), (30.61920331726396, 6172404), (30.124736384916716, 4857055), (29.95739605589631, 5559530), (29.91600536432348, 5406347), (29.896227397992213, 1845958)]}, {'query_id': '88495', 'scores': [(15.077352965850473, 1091557), (15.046667342820506, 2875829), (15.017734827039696, 6579243), (14.99496516252983, 6846565), (14.99496516252983, 1667678), (14.977151395198625, 7950301), (14.977151395198625, 5897696), (14.974296984491962, 5915611), (14.926849591670678, 5459872), (14.925449214907948, 3668359)]}, {'query_id': '911232', 'scores': [(48.94103136051328, 8816622), (46.601150696209956, 2999930), (45.74537563176936, 2999933), (44.351854430231576, 8816620), (42.68722004445147, 6928655), (41.57924920747766, 1879470), (39.374883445396314, 2999925), (37.829807672599586, 8816626), (37.4224271939379, 2999926), (36.80304716499916, 2999929)]}, {'query_id': '914916', 'scores': [(24.115066183987224, 6042350), (24.115066183987224, 2631739), (23.900148596496386, 2631743), (23.702185841976544, 7040507), (23.58277717125479, 8520449), (23.58277717125479, 8481901), (23.58277717125479, 5177082), (23.58277717125479, 2001628), (23.58277717125479, 2001626), (23.5243453755621, 6071696)]}, {'query_id': '91576', 'scores': [(21.626661335749745, 8279891), (21.254055200294314, 3780281), (21.186062669141794, 1382984), (20.30820885644591, 4161753), (16.997766322096204, 3540074), (16.798465365922368, 6805821), (16.798465365922368, 6137131), (16.798465365922368, 3255380), (16.42585923046694, 4066736), (16.42585923046694, 3880194)]}, {'query_id': '918162', 'scores': [(37.736907364185775, 2692389), (37.736907364185775, 2692384), (37.647986597970856, 2692388), (36.78845626874412, 267612), (36.12998838129908, 8217238), (35.36692533963837, 4658366), (35.36692533963837, 1290157), (35.36692533963837, 672942), (34.032798311938706, 2334496), (32.285989138142504, 1223048)]}, {'query_id': '938400', 'scores': [(25.88486975747914, 7239123), (24.674026489890096, 6499450), (24.674026489890096, 2969902), (22.757135958437825, 2870510), (22.49427807091705, 3074064), (21.827023433364314, 7029656), (21.827023433364314, 3424110), (21.827023433364314, 2371820), (21.827023433364314, 790219), (21.827023433364314, 47285)]}, {'query_id': '940547', 'scores': [(31.500445897881193, 4481817), (30.889070109606976, 6404591), (30.20682517048933, 4664999), (29.59174492765908, 3822464), (29.582790308462954, 4820847), (29.13328556933979, 7793152), (29.13328556933979, 4100051), (29.13328556933979, 3408094), (28.957481695636652, 1976277), (28.237017424336784, 2947470)]}, {'query_id': '945835', 'scores': [(28.60197933240492, 4431195), (27.78575375488353, 4539379), (27.670557836323006, 5548718), (27.670557836323006, 4431193), (27.43742163126144, 4425412), (27.43742163126144, 61200), (27.43742163126144, 61192), (27.157326309060814, 5548713), (27.157326309060814, 4539378), (26.861507015064547, 61193)]}, {'query_id': '978031', 'scores': [(19.435170985035278, 8744912), (19.144578836480417, 8744911), (18.710755081464356, 4755047), (18.702871910179383, 8744914), (18.14379076161408, 261672), (18.078843186159986, 8802560), (17.99327733278395, 3698979), (17.87542247813394, 5287819), (17.87542247813394, 4543384), (17.87542247813394, 2398979)]}, {'query_id': '985594', 'scores': [(19.67035606397626, 1657406), (19.39588597936264, 8677959), (19.39588597936264, 8677953), (19.39588597936264, 7740126), (19.39588597936264, 4286031), (19.39588597936264, 1657408), (19.39588597936264, 1657405), (19.39588597936264, 655890), (18.95507038892258, 7740128), (18.95507038892258, 7740125)]}, {'query_id': '99005', 'scores': [(45.2135054158107, 8257322), (44.35849888085889, 6493343), (43.32028239431068, 6493344), (43.145437455444196, 2875400), (43.145437455444196, 1375045), (41.83955027819647, 297028), (41.83955027819647, 297023), (39.153255240284125, 6327640), (39.06156508279107, 8257320), (38.66518814029489, 8476970)]}, {'query_id': '997622', 'scores': [(27.76902319322393, 4543865), (27.10310266495629, 73465), (25.929397523008838, 4518222), (25.929397523008838, 2275574), (25.710176817953887, 73459), (25.41268901627689, 4558331), (25.080989737030873, 4643397), (24.23898387432944, 8533344), (24.23898387432944, 3913148), (23.86791654571513, 5430569)]}, {'query_id': '999466', 'scores': [(22.22063876135589, 8748919), (22.22063876135589, 8748914), (20.771466668223987, 8748922), (20.771466668223987, 8748918), (18.3747589757366, 8748921), (18.3747589757366, 8748920), (18.3747589757366, 8748916), (18.3747589757366, 8748915), (18.3747589757366, 8748913)]}, {'query_id': '132622', 'scores': [(23.01604771891932, 5901873), (22.71211571786607, 962815), (22.047486249068534, 2459821), (22.047486249068534, 821354), (21.95264894655249, 2979805), (21.328911375048648, 7122663), (21.039073000281075, 5656719), (20.9665369265526, 3162996), (20.666644970146464, 6358747), (20.36034990519786, 3358937)]}]\n"
          ]
        }
      ],
      "source": [
        "tfidf_results = query_processing(trec_dl_2020.queries_iter(), taat_tfidf)\n",
        "print(tfidf_results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QW_f8sdCSu6J"
      },
      "source": [
        "The results obtained on Google Colab in this section exhibit significant variability, primarily due to the underlying machines used by the platform. During our experiments, we observed approximate processing times for each query method: 318 seconds for DAAT-BM25 (1.59s/query), 145 seconds for TAAT-BM25 (0.73s/query), 306 seconds for DAAT-TFIDF (1.53s/query), and 136 seconds for TAAT-TFIDF (0.68s/query). However, these timings are highly dependent on the specific hardware allocated by Colab at the time of execution."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bpb3m8ELHcdo"
      },
      "source": [
        "# 5. Evaluation with TREC-style measures\n",
        "To evaluate retrieval performance, we use the TREC evaluation method with `ir_measures`.\n",
        "\n",
        "This section generates a run file and QRELs for the TREC evaluation tool."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i_cJcwV1Hcdo",
        "outputId": "e1d13073-cb1f-4926-f40a-612407f3b3c5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "GenericQuery(query_id='1030303', text='who is aziz hashim')\n",
            "GenericQuery(query_id='1037496', text='who is rep scalise?')\n",
            "GenericQuery(query_id='1043135', text='who killed nicholas ii of russia')\n"
          ]
        }
      ],
      "source": [
        "for query in list(trec_dl_2020.queries_iter())[:3]:\n",
        "    print(query)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GhrquNSbHcdp",
        "outputId": "b8022f9c-87dd-408e-aa7f-817763ffe76a"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[INFO] [starting] https://trec.nist.gov/data/deep/2020qrels-pass.txt\n",
            "[INFO] [finished] https://trec.nist.gov/data/deep/2020qrels-pass.txt: [00:00] [107B] [776kB/s]\n",
            "[WARNING] Download failed: Expected md5 hash to be 0355ccee7509ac0463e8278186cdd8d1 but got ff4ed829ffd04e9c6095842b07d11f09\n",
            "[INFO] [starting] https://mirror.ir-datasets.com/0355ccee7509ac0463e8278186cdd8d1\n",
            "[INFO] [finished] https://mirror.ir-datasets.com/0355ccee7509ac0463e8278186cdd8d1: [00:00] [219kB] [6.40MB/s]\n",
            "                                                                                           "
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TrecQrel(query_id='23849', doc_id='1020327', relevance=2, iteration='0')\n",
            "TrecQrel(query_id='23849', doc_id='1034183', relevance=3, iteration='0')\n",
            "TrecQrel(query_id='23849', doc_id='1120730', relevance=0, iteration='0')\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\r"
          ]
        }
      ],
      "source": [
        "for qrel in list(trec_dl_2020.qrels_iter())[:3]:\n",
        "  print(qrel)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BfrhOcCVrwjC"
      },
      "source": [
        "## 5.1 Run File generation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "HT-n8O0aHcdp"
      },
      "outputs": [],
      "source": [
        "def generate_run(results):\n",
        "    \"\"\"\n",
        "    Generate a TREC-formatted run list from query results.\n",
        "\n",
        "    Args:\n",
        "        results (list[dict]): A list of query results, where each result contains:\n",
        "            - `query_id` (int): The ID of the query.\n",
        "            - `scores` (list[tuple[float, int]]): A list of (score, doc_id) tuples.\n",
        "\n",
        "    Returns:\n",
        "        list[str]: A list of strings formatted in TREC run format.\n",
        "    \"\"\"\n",
        "\n",
        "    trec_run_list = []                      # List to store TREC-formatted lines\n",
        "\n",
        "    for doc_scores in results:              # Iterate over each query result\n",
        "        rank = 1\n",
        "        query_id = doc_scores['query_id']\n",
        "        scores = doc_scores['scores']\n",
        "\n",
        "        for score, doc_id in scores:\n",
        "            # Format the result as a TREC-compliant line\n",
        "            line = f\"{query_id} Q0 {doc_id} {rank} {score} GOODFELLAS\"\n",
        "            trec_run_list.append(line)\n",
        "            rank += 1\n",
        "\n",
        "    return trec_run_list\n",
        "\n",
        "# Generate TREC-formatted run lists for BM25 and TF-IDF results\n",
        "trec_bm25_run_list = generate_run(bm25_results)\n",
        "trec_tfidf_run_list = generate_run(tfidf_results)\n",
        "\n",
        "# Write the BM25 run list to a TREC-eval compatible file\n",
        "with open(\"trec_eval_bm25_run_file.txt\", \"w\") as f:\n",
        "    for line in trec_bm25_run_list:\n",
        "        f.write(line + \"\\n\")\n",
        "\n",
        "# Write the TF-IDF run list to a separate TREC-eval compatible file\n",
        "with open(\"trec_eval_tfidf_run_file.txt\", \"w\") as f:\n",
        "    for line in trec_tfidf_run_list:\n",
        "        f.write(line + \"\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RGO-qvpLrwjH"
      },
      "source": [
        "## 5.2 Qrels File generation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "BbLdcjrCHcdp"
      },
      "outputs": [],
      "source": [
        "qrels_file = []     # List to store lines formatted for TREC-Eval qrels\n",
        "\n",
        "# Iterate over qrels data provided by the TREC DL 2020 dataset\n",
        "for qrel in trec_dl_2020.qrels_iter():\n",
        "    # Format the qrel information as per TREC-Eval requirements\n",
        "    # Format: <query_id> 0 <doc_id> <relevance>\n",
        "    line = f\"{qrel.query_id} 0 {qrel.doc_id} {qrel.relevance}\"\n",
        "    qrels_file.append(line)\n",
        "\n",
        "# Write the qrels list to a file in TREC-Eval compatible format\n",
        "with open(\"trec_eval_qrels_file.txt\", \"w\") as f:\n",
        "    for line in qrels_file:\n",
        "        f.write(line + \"\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jWj3BsAMrwjH"
      },
      "source": [
        "## 5.3 Results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "zWWbnIHvHcdp"
      },
      "outputs": [],
      "source": [
        "# Define evaluation measures for the retrieval models\n",
        "measures = [\n",
        "    P@5,              # Precision at rank 5\n",
        "    P(rel=2)@5,       # Precision at rank 5, considering relevance level >= 2\n",
        "    nDCG@10,          # Normalized Discounted Cumulative Gain at rank 10\n",
        "    AP,               # Average Precision\n",
        "    AP(rel=2),        # Average Precision, considering relevance level >= 2\n",
        "    Bpref,            # Binary preference\n",
        "    Bpref(rel=2),     # Binary preference, considering relevance level >= 2\n",
        "    Judged@10         # Fraction of top 10 documents that were judged\n",
        "]\n",
        "\n",
        "# Load qrels (ground truth relevance judgments)\n",
        "qrels = ir_measures.read_trec_qrels('trec_eval_qrels_file.txt')\n",
        "\n",
        "# Evaluate BM25 results using the defined measures\n",
        "bm25_run = ir_measures.read_trec_run('trec_eval_bm25_run_file.txt')\n",
        "bm25_results = ir_measures.calc_aggregate(measures, qrels, bm25_run)\n",
        "\n",
        "qrels = ir_measures.read_trec_qrels('trec_eval_qrels_file.txt')\n",
        "\n",
        "# Evaluate TF-IDF results using the same qrels and measures\n",
        "tfidf_run = ir_measures.read_trec_run('trec_eval_tfidf_run_file.txt')\n",
        "tfidf_results = ir_measures.calc_aggregate(measures, qrels, tfidf_run)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xWlPxADGrwjH",
        "outputId": "4bdb9fb0-7f5a-45b2-9b0e-36349852963b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                  BM25    TF-IDF\n",
            "P@5           0.607407  0.607407\n",
            "AP(rel=2)     0.176679  0.176508\n",
            "P(rel=2)@5    0.403704  0.403704\n",
            "Bpref         0.156412  0.156832\n",
            "AP            0.140604  0.141106\n",
            "Bpref(rel=2)  0.197055  0.196541\n",
            "nDCG@10       0.478191  0.478917\n",
            "Judged@10     0.925926  0.925926\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Create a DataFrame to compare BM25 and TF-IDF evaluation metrics\n",
        "# Each column represents the results for a retrieval model\n",
        "df = pd.DataFrame({\n",
        "    \"BM25\": bm25_results,\n",
        "    \"TF-IDF\": tfidf_results\n",
        "})\n",
        "\n",
        "print(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qg_uFHDOSu6K"
      },
      "source": [
        "Both BM25 and TF-IDF show nearly identical performance, with minimal metric variations. BM25 slightly outperforms TF-IDF on AP(rel=2) and Bpref(rel=2), while TF-IDF yields marginally higher AP and nDCG@10. Precision at rank 5 (P@5) and judged coverage (Judged@10) remain the same for both, indicating very similar retrieval effectiveness overall."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bxOI4RkIrwjH"
      },
      "source": [
        "## 5.4 Comparison with PyTerrier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qn0eky0LZuWl"
      },
      "source": [
        "The same set of queries has been evaluated using PyTerrier, which serves as benchmark, and a comparison between the latter and results obtained  through the implementation seen so far has been performed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 132
        },
        "id": "T5HqAq-FrwjH",
        "outputId": "fbe318fa-e00a-4690-edb7-81a0f3044455"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \")\",\n  \"rows\": 2,\n  \"fields\": [\n    {\n      \"column\": \"name\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"TerrierRetr(BM25)\",\n          \"TerrierRetr(TF_IDF)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"P@5\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0,\n        \"min\": 0.6259259259259259,\n        \"max\": 0.6259259259259259,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0.6259259259259259\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"P(rel=2)@5\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0,\n        \"min\": 0.3925925925925925,\n        \"max\": 0.3925925925925925,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0.3925925925925925\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"nDCG@10\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0007440302222365624,\n        \"min\": 0.49257516071215307,\n        \"max\": 0.4936273783432555,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.4936273783432555\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AP\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.00046121868230354087,\n        \"min\": 0.35807197489957543,\n        \"max\": 0.35872423661530894,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.35872423661530894\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AP(rel=2)\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0003109810406316397,\n        \"min\": 0.29254805937899814,\n        \"max\": 0.2929878529843003,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0.2929878529843003\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Judged@10\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0,\n        \"min\": 0.9722222222222221,\n        \"max\": 0.9722222222222221,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0.9722222222222221\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-ad740d6b-f0fa-47f3-a73e-4ca5a0867e25\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>name</th>\n",
              "      <th>P@5</th>\n",
              "      <th>P(rel=2)@5</th>\n",
              "      <th>nDCG@10</th>\n",
              "      <th>AP</th>\n",
              "      <th>AP(rel=2)</th>\n",
              "      <th>Judged@10</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>TerrierRetr(TF_IDF)</td>\n",
              "      <td>0.625926</td>\n",
              "      <td>0.392593</td>\n",
              "      <td>0.492575</td>\n",
              "      <td>0.358072</td>\n",
              "      <td>0.292548</td>\n",
              "      <td>0.972222</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>TerrierRetr(BM25)</td>\n",
              "      <td>0.625926</td>\n",
              "      <td>0.392593</td>\n",
              "      <td>0.493627</td>\n",
              "      <td>0.358724</td>\n",
              "      <td>0.292988</td>\n",
              "      <td>0.972222</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ad740d6b-f0fa-47f3-a73e-4ca5a0867e25')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ad740d6b-f0fa-47f3-a73e-4ca5a0867e25 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ad740d6b-f0fa-47f3-a73e-4ca5a0867e25');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-f32eada5-48dd-4e21-8b1f-2249f6b1e021\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-f32eada5-48dd-4e21-8b1f-2249f6b1e021')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-f32eada5-48dd-4e21-8b1f-2249f6b1e021 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                  name       P@5  P(rel=2)@5   nDCG@10        AP  AP(rel=2)  \\\n",
              "0  TerrierRetr(TF_IDF)  0.625926    0.392593  0.492575  0.358072   0.292548   \n",
              "1    TerrierRetr(BM25)  0.625926    0.392593  0.493627  0.358724   0.292988   \n",
              "\n",
              "   Judged@10  \n",
              "0   0.972222  \n",
              "1   0.972222  "
            ]
          },
          "execution_count": 32,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from pyterrier.measures import P, nDCG, AP, Judged\n",
        "\n",
        "# Load the MSMARCO Passage Retrieval dataset\n",
        "dataset = pt.get_dataset('msmarco_passage')\n",
        "\n",
        "# Run an experiment comparing TF-IDF and BM25 retrieval models\n",
        "pt.Experiment(\n",
        "    [\n",
        "        # TF-IDF retriever from the Terrier index\n",
        "        pt.terrier.Retriever.from_dataset('msmarco_passage', 'terrier_stemmed', wmodel='TF_IDF'),\n",
        "        # BM25 retriever from the Terrier index\n",
        "        pt.terrier.Retriever.from_dataset('msmarco_passage', 'terrier_stemmed', wmodel='BM25'),\n",
        "    ],\n",
        "    dataset.get_topics('test-2020'),                                    # Test topics for the experiment\n",
        "    dataset.get_qrels('test-2020'),                                     # Ground truth relevance judgments\n",
        "    eval_metrics=[P@5, P(rel=2)@5, nDCG@10, AP, AP(rel=2), Judged@10],  # Evaluation metrics\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7hMOJUE-ZuWm"
      },
      "source": [
        "Despite PyTerrier’s overall superior performance in terms, the proposed custom implementation achieved competitive results in certain scenarios:\n",
        "\n",
        "- **nDCG@10**: achieved scores of 0.4782 for BM25 and 0.4789 for TF-IDF, closely approaching PyTerrier's 0.4936 and 0.4926, respectively.\n",
        "- **BM25 Precision (P@5)**: attained a precision of 0.607, only slightly lower than PyTerrier’s 0.626, demonstrating competitive ranking in the top 5 results.\n",
        "- **Judged@10**: achieved a score of 0.926, which is relatively close to PyTerrier’s 0.972.\n",
        "\n",
        "These results highlight the potential of the custom implementation, especially considering it was developed from scratch without the extensive optimization and tuning present in PyTerrier, which for instance outperformed in  **Average Precision** achieving 0.358 compared to 0.141 for the custom implementation, highlighting the impact of advanced tuning and indexing on retrieval accuracy."
      ]
    }
  ],
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "gpuType": "V5E1",
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}