01-Load-Data-ACogSearch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Welcome to this repository. We will be walking you to a series of notebooks in which you will understand how Agents and RAG works (Retrieval Augmented Generation, a technique that combines the power of search and generative AI to answer user queries). We will work with different sources (Azure AI Search, Files, SQL Server, Websites, APIs, etc) and at the end of the notebooks you will understand why the magic happens with the combination of:\n",
    "\n",
    "1) Multi-Agents: Agents talking to each other\n",
    "2) Azure OpenAI models\n",
    "3) Very detailed prompts\n",
    "\n",
    "But we need to start from the basics, so let's begin with Azure AI Search and how it works:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load and Enrich multiple file types Azure AI Search\n",
    "\n",
    "In this Jupyter Notebook, we create and run enrichment steps to unlock searchable content in the specified Azure blob. It performs operations over mixed content in Azure Storage, such as images and application files, using a skillset that analyzes and extracts text information that becomes searchable in Azure Cognitive Search. \n",
    "The reference sample can be found at [Tutorial: Use Python and AI to generate searchable content from Azure blobs](https://docs.microsoft.com/azure/search/cognitive-search-tutorial-blob-python).\n",
    "\n",
    "In this demo we are going to be using a private (so we can mimic a private data lake scenario) Blob Storage container that has all the dialogues of each episode of the TV Series show: FRIENDS. (3.1k text files).\n",
    "\n",
    "Although only  TXT files are used here, this can be done at a much larger scale and Azure Cognitive Search supports a range of other file formats including: PDF, Microsoft Office (DOCX/DOC, XSLX/XLS, PPTX/PPT, MSG), HTML, XML, ZIP, and plain text files (including JSON).\n",
    "Azure Search support the following sources: [Data Sources Gallery](https://learn.microsoft.com/EN-US/AZURE/search/search-data-sources-gallery)\n",
    "\n",
    "This notebook creates the following objects on your search service:\n",
    "\n",
    "+ data source\n",
    "+ search index\n",
    "+ skillset\n",
    "+ indexer\n",
    "\n",
    "This notebook calls the [Search REST APIs](https://docs.microsoft.com/rest/api/searchservice/), but you can also use the Azure.Search.Documents client library in the Azure SDK for Python to perform the same steps. See this [Python quickstart](https://docs.microsoft.com/azure/search/search-get-started-python) for details.\n",
    "\n",
    "To run this notebook, you should have already created the Azure services on README. Once you've done this, you can run all cells, but the query won't return results until the indexer is finished and the search index is loaded. \n",
    "\n",
    "We recommend running each step and making sure it completes before moving on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![cog-search](./images/Cog-Search-Enrich.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import shutil\n",
    "import requests\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "from common.utils import upload_file_to_blob, extract_zip_file, upload_directory_to_blob\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the names for the data source, skillset, index and indexer\n",
    "datasource_name = \"srch-datasource-files\"\n",
    "index_name = \"srch-index-files\"\n",
    "skillset_name = \"srch-skillset-files\"\n",
    "indexer_name = \"srch-indexer-files\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Setup the Payloads header\n",
    "headers = {'Content-Type': 'application/json','api-key': os.environ['AZURE_SEARCH_KEY']}\n",
    "params = {'api-version': os.environ['AZURE_SEARCH_API_VERSION']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload local dataset to Blob Container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define connection string and other parameters\n",
    "BLOB_CONTAINER_NAME = \"friends\"\n",
    "BLOB_NAME = \"friends_transcripts.zip\"\n",
    "LOCAL_FILE_PATH = \"./data/\" + BLOB_NAME  # Path to the local file you want to upload\n",
    "upload_directory = \"./data/temp_extract\"  # Temporary directory to extract the zip file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This process below takes about 11 mins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./data/friends_transcripts.zip ... \n",
      "Extracted ./data/friends_transcripts.zip to ./data/temp_extract\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Uploading Files: 100%|██████████████████████████████████████████| 3107/3107 [08:53<00:00,  5.83it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Temp Folder: ./data/temp_extract removed\n",
      "CPU times: user 31.6 s, sys: 5.19 s, total: 36.8 s\n",
      "Wall time: 11min\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Extract the zip file\n",
    "extract_zip_file(LOCAL_FILE_PATH, upload_directory)\n",
    "\n",
    "# Upload the extracted files and folder structure\n",
    "upload_directory_to_blob(upload_directory, BLOB_CONTAINER_NAME)\n",
    "\n",
    "# Clean up: Optionally, you can remove the temp folder after uploading\n",
    "shutil.rmtree(upload_directory)\n",
    "print(f\"Temp Folder: {upload_directory} removed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Data Source (Blob container with the Arxiv CS pdfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# The following code sends the json paylod to Azure Search engine to create the Datasource\n",
    "\n",
    "datasource_payload = {\n",
    "    \"name\": datasource_name,\n",
    "    \"description\": \"Demo files to demonstrate ai search capabilities.\",\n",
    "    \"type\": \"azureblob\",\n",
    "    \"credentials\": {\n",
    "        \"connectionString\": os.environ['BLOB_CONNECTION_STRING']\n",
    "    },\n",
    "    \"dataDeletionDetectionPolicy\" : {\n",
    "        \"@odata.type\" :\"#Microsoft.Azure.Search.NativeBlobSoftDeleteDeletionDetectionPolicy\" # this makes sure that if the item is deleted from the source, it will be deleted from the index\n",
    "    },\n",
    "    \"container\": {\n",
    "        \"name\": BLOB_CONTAINER_NAME\n",
    "    }\n",
    "}\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/datasources/\" + datasource_name,\n",
    "                 data=json.dumps(datasource_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 201 - Successfully created\n",
    "- 204 - Succesfully overwritten\n",
    "- 40X - Authentication Error\n",
    "\n",
    "For information on Change and Delete file detection please see [HERE](https://learn.microsoft.com/en-us/azure/search/search-howto-index-changed-deleted-blobs?tabs=rest-api)\n",
    "\n",
    "Also, if your data is on AWS or GCP, and do not want to move it to Azure, you can create a Azure Fabric shortcut in OneLake, and use Fabric as a datasource here. From the documentation [HERE](https://learn.microsoft.com/en-us/azure/search/search-how-to-index-onelake-files):\n",
    "> If you use Microsoft Fabric and OneLake for data access to Amazon Web Services (AWS) and Google data sources, use this indexer to import external data into a search index. This indexer is available through the Azure portal, the 2024-05-01-preview REST API, and Azure SDK beta packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# If you have a 403 code, probably you have a wrong endpoint or key, you can debug by uncomment this\n",
    "# r.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Azure AI Search, a search index is your searchable content, available to the search engine for indexing, full text search, vector search, hybrid search, and filtered queries. An index is defined by a schema and saved to the search service, with data import following as a second step. This content exists within your search service, apart from your primary data stores, which is necessary for the millisecond response times expected in modern search applications. Except for indexer-driven indexing scenarios, the search service never connects to or queries your source data.\n",
    "\n",
    "Reference:\n",
    "\n",
    "https://learn.microsoft.com/en-us/azure/search/search-what-is-an-index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice below how we are creating a vector store. In Azure AI Search, a vector store has an index schema that defines vector and nonvector fields, a vector configuration for algorithms that create the embedding space, and settings on vector field definitions that are used in query requests. \n",
    "\n",
    "We are also setting a semantic ranking over a result set, promoting the most semantically relevant results to the top of the stack. You can also get semantic captions, with highlights over the most relevant terms and phrases, and semantic answers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A note about compression and space optimization**: \n",
    "Creating a Vector Index requires optimization otherwise it can get very large and very expensive very quickly. From the documentation:\n",
    "> Embeddings, or the numerical representation of heterogeneous content, are the basis of vector search workloads, but the sizes of embeddings make them hard to scale and expensive to process. Significant research and productization have produced multiple solutions for improving scale and reducing processing times. Azure AI Search taps into a number these capabilities for faster and cheaper vector workloads.\n",
    "\n",
    "\n",
    "Below we will implement some of these compression techniques when it says `Compression (optional)`.\n",
    "For detailed information about compression techniques please check the documentation [HERE](https://learn.microsoft.com/en-us/azure/search/vector-search-index-size?tabs=portal-vector-quota)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create an index\n",
    "# Queries operate over the searchable fields and filterable fields in the index\n",
    "index_payload = {\n",
    "    \"name\": index_name,\n",
    "    \"vectorSearch\": {\n",
    "        \"algorithms\": [\n",
    "            {\n",
    "                \"name\": \"use-hnsw\",\n",
    "                \"kind\": \"hnsw\",\n",
    "            }\n",
    "        ],\n",
    "        \"compressions\": [ # Compression (optional)\n",
    "            {\n",
    "                \"name\": \"use-scalar\",\n",
    "                \"kind\": \"scalarQuantization\",\n",
    "                \"rescoringOptions\": {\n",
    "                    \"enableRescoring\": \"true\",\n",
    "                    \"defaultOversampling\": 10,\n",
    "                    \"rescoreStorageMethod\": \"preserveOriginals\"\n",
    "                },\n",
    "                \"scalarQuantizationParameters\": {\n",
    "                    \"quantizedDataType\": \"int8\"\n",
    "                },\n",
    "                \"truncationDimension\": 1024\n",
    "            },\n",
    "            {\n",
    "                \"name\": \"use-binary\",\n",
    "                \"kind\": \"binaryQuantization\",\n",
    "                \"rescoringOptions\": {\n",
    "                    \"enableRescoring\": \"true\",\n",
    "                    \"defaultOversampling\": 10,\n",
    "                    \"rescoreStorageMethod\": \"preserveOriginals\"\n",
    "                },\n",
    "                \"truncationDimension\": 1024\n",
    "            }\n",
    "        ],\n",
    "        \"vectorizers\": [ # converts text (or images) to vectors during query execution.\n",
    "            {\n",
    "                \"name\": \"use-openai\",\n",
    "                \"kind\": \"azureOpenAI\",\n",
    "                \"azureOpenAIParameters\":\n",
    "                {\n",
    "                    \"resourceUri\" : os.environ['AZURE_OPENAI_ENDPOINT'],\n",
    "                    \"apiKey\" : os.environ['AZURE_OPENAI_API_KEY'],\n",
    "                    \"deploymentId\" : os.environ['EMBEDDING_DEPLOYMENT_NAME'],\n",
    "                    \"modelName\" : os.environ['EMBEDDING_DEPLOYMENT_NAME'],\n",
    "     \n",
    "                }\n",
    "            }\n",
    "        ],\n",
    "        \"profiles\": [\n",
    "           {\n",
    "                \"name\": \"vector-profile-hnsw-scalar\",\n",
    "                \"compression\": \"use-scalar\", # Compression (optional)\n",
    "                \"algorithm\": \"use-hnsw\",\n",
    "                \"vectorizer\": \"use-openai\"\n",
    "           },\n",
    "           {\n",
    "                \"name\": \"vector-profile-hnsw-binary\",\n",
    "                \"compression\": \"use-binary\",\n",
    "                \"algorithm\": \"use-hnsw\",\n",
    "                \"vectorizer\": \"use-openai\"\n",
    "           }\n",
    "         ]\n",
    "    },\n",
    "    \"semantic\": {\n",
    "        \"configurations\": [\n",
    "            {\n",
    "                \"name\": \"my-semantic-config\",\n",
    "                \"prioritizedFields\": {\n",
    "                    \"titleField\": {\n",
    "                        \"fieldName\": \"title\"\n",
    "                    },\n",
    "                    \"prioritizedContentFields\": [\n",
    "                        {\n",
    "                            \"fieldName\": \"chunk\"\n",
    "                        }\n",
    "                    ],\n",
    "                    \"prioritizedKeywordsFields\": []\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    \"fields\": [\n",
    "        {\"name\": \"id\", \"type\": \"Edm.String\", \"key\": \"true\", \"analyzer\": \"keyword\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\",\"facetable\": \"false\"},\n",
    "        {\"name\": \"ParentKey\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"title\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"name\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"location\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},   \n",
    "        {\"name\": \"chunk\",\"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\n",
    "            \"name\": \"chunkVector\",\n",
    "            \"type\": \"Collection(Edm.Half)\", # Compression (optional)\n",
    "            \"dimensions\": 3072, # IMPORTANT: Make sure these dimmensions match your embedding model name\n",
    "            \"vectorSearchProfile\": \"vector-profile-hnsw-scalar\",\n",
    "            \"searchable\": \"true\",\n",
    "            \"retrievable\": \"false\",\n",
    "            \"filterable\": \"false\",\n",
    "            \"sortable\": \"false\",\n",
    "            \"facetable\": \"false\",\n",
    "            \"stored\": \"false\" # Compression (optional)\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + index_name,\n",
    "                 data=json.dumps(index_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# r.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Semantic Search capabilities\n",
    "As you can see above in the index payload, there is a `semantic configuration`. What is that?\n",
    "\n",
    "Semantic ranker is a collection of query-related capabilities that improve the quality of an initial BM25-ranked or RRF-ranked search result for text-based queries. When you enable it on your search service, semantic ranking extends the query execution pipeline in two ways:\n",
    "\n",
    "    First, it adds secondary ranking over an initial result set that was scored using BM25 or RRF. This secondary ranking uses multi-lingual, deep learning models adapted from Microsoft Bing to promote the most semantically relevant results.\n",
    "\n",
    "    Second, it extracts and returns captions and answers in the response, which you can render on a search page to improve the user's search experience.\n",
    "\n",
    "\n",
    "For deeper explanation and limitations see [HERE](https://learn.microsoft.com/en-us/azure/search/semantic-ranking)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Skillset - OCR, Text Splitter, AzureOpenAIEmbeddingSkill\n",
    "\n",
    "We need to create now the skillset. This is a set of steps in which we use AI Services to enrich the documents by extracting information, applying OCR, splitting, and embedding chunks, among other skills.\n",
    "\n",
    "https://learn.microsoft.com/en-us/azure/search/cognitive-search-working-with-skillsets\n",
    "\n",
    "https://learn.microsoft.com/en-us/azure/search/cognitive-search-predefined-skills\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice below that we are using IndexProjections. By default, one document processed within a skillset maps to a single document in the search index. This means that if you perform chunking of an input text and then perform enrichments on each chunk, the result in the index when mapped via outputFieldMappings is an array of the generated enrichments. **With index projections, you define a context at which to map each chunk of enriched data to its own search document**. This allows you to apply a one-to-many mapping of a document's enriched data to the search index.\n",
    "    \n",
    "The parameter: `\"projectionMode\": \"skipIndexingParentDocuments\"` allows us to skip the indexing of the parent documents, and keep only the index with the chunks and its vectors.\n",
    "\n",
    "### Content Lifecycle\n",
    "If the indexer data source supports change tracking and deletion detection, the indexing process can synchronize the primary (parend documents) and secondary indexes (chunks) to pick up those changes.\n",
    "\n",
    "Each time you run the indexer and skillset, the index projections are updated if the skillset or underlying source data has changed. Any changes picked up by the indexer are propagated through the enrichment process to the projections in the index, ensuring that your projected data is a current representation of content in the originating data source. This will save you weeks of programming and a lot of headaches trying to keep the content in sync."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create a skillset\n",
    "skillset_payload = {\n",
    "    \"name\": skillset_name,\n",
    "    \"description\": \"e2e Skillset for RAG - Files\",\n",
    "    \"skills\":\n",
    "    [\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Vision.OcrSkill\",\n",
    "            \"description\": \"Extract text (plain and structured) from image.\",\n",
    "            \"context\": \"/document/normalized_images/*\",\n",
    "            \"defaultLanguageCode\": \"en\",\n",
    "            \"detectOrientation\": True,\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                  \"name\": \"image\",\n",
    "                  \"source\": \"/document/normalized_images/*\"\n",
    "                }\n",
    "            ],\n",
    "                \"outputs\": [\n",
    "                {\n",
    "                  \"name\": \"text\",\n",
    "                  \"targetName\" : \"images_text\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.MergeSkill\",\n",
    "            \"description\": \"Create merged_text, which includes all the textual representation of each image inserted at the right location in the content field. This is useful for PDF and other file formats that supported embedded images.\",\n",
    "            \"context\": \"/document\",\n",
    "            \"insertPreTag\": \" \",\n",
    "            \"insertPostTag\": \" \",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                  \"name\":\"text\", \"source\": \"/document/content\"\n",
    "                },\n",
    "                {\n",
    "                  \"name\": \"itemsToInsert\", \"source\": \"/document/normalized_images/*/images_text\"\n",
    "                },\n",
    "                {\n",
    "                  \"name\":\"offsets\", \"source\": \"/document/normalized_images/*/contentOffset\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                  \"name\": \"mergedText\", \n",
    "                  \"targetName\" : \"merged_text\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.SplitSkill\",\n",
    "            \"context\": \"/document\",\n",
    "            \"textSplitMode\": \"pages\",  # although it says \"pages\" it actally means chunks, not actual pages\n",
    "            \"maximumPageLength\": 5000, # 5000 characters is default and a good choice\n",
    "            \"pageOverlapLength\": 750,  # 15% overlap among chunks\n",
    "            \"defaultLanguageCode\": \"en\",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                    \"name\": \"text\",\n",
    "                    \"source\": \"/document/merged_text\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                    \"name\": \"textItems\",\n",
    "                    \"targetName\": \"chunks\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill\",\n",
    "            \"description\": \"Azure OpenAI Embedding Skill\",\n",
    "            \"context\": \"/document/chunks/*\",\n",
    "            \"resourceUri\": os.environ['AZURE_OPENAI_ENDPOINT'],\n",
    "            \"apiKey\": os.environ['AZURE_OPENAI_API_KEY'],\n",
    "            \"deploymentId\": os.environ['EMBEDDING_DEPLOYMENT_NAME'],\n",
    "            \"modelName\": os.environ['EMBEDDING_DEPLOYMENT_NAME'],\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                    \"name\": \"text\",\n",
    "                    \"source\": \"/document/chunks/*\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                    \"name\": \"embedding\",\n",
    "                    \"targetName\": \"vector\"\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    \"indexProjections\": {\n",
    "        \"selectors\": [\n",
    "            {\n",
    "                \"targetIndexName\": index_name,\n",
    "                \"parentKeyFieldName\": \"ParentKey\",\n",
    "                \"sourceContext\": \"/document/chunks/*\",\n",
    "                \"mappings\": [\n",
    "                    {\n",
    "                        \"name\": \"title\",\n",
    "                        \"source\": \"/document/title\"\n",
    "                    },\n",
    "                    {\n",
    "                        \"name\": \"name\",\n",
    "                        \"source\": \"/document/name\"\n",
    "                    },\n",
    "                    {\n",
    "                        \"name\": \"location\",\n",
    "                        \"source\": \"/document/location\"\n",
    "                    },\n",
    "                    {\n",
    "                        \"name\": \"chunk\",\n",
    "                        \"source\": \"/document/chunks/*\"\n",
    "                    },\n",
    "                    {\n",
    "                        \"name\": \"chunkVector\",\n",
    "                        \"source\": \"/document/chunks/*/vector\"\n",
    "                    }\n",
    "                ]\n",
    "            }\n",
    "        ],\n",
    "        \"parameters\": {\n",
    "            \"projectionMode\": \"skipIndexingParentDocuments\"\n",
    "        }\n",
    "    },\n",
    "    \"cognitiveServices\": {\n",
    "        \"@odata.type\": \"#Microsoft.Azure.Search.CognitiveServicesByKey\",\n",
    "        \"description\": os.environ['COG_SERVICES_NAME'],\n",
    "        \"key\": os.environ['COG_SERVICES_KEY']\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/skillsets/\" + skillset_name,\n",
    "                 data=json.dumps(skillset_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create and Run the Indexer - (runs the pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three components you have created thus far (data source, skillset, index) are inputs to an indexer. Creating the indexer on Azure Cognitive Search is the event that puts the entire pipeline into motion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create an indexer\n",
    "indexer_payload = {\n",
    "    \"name\": indexer_name,\n",
    "    \"dataSourceName\": datasource_name,\n",
    "    \"targetIndexName\": index_name,\n",
    "    \"skillsetName\": skillset_name,\n",
    "    \"schedule\" : { \"interval\" : \"PT30M\"}, # How often do you want to check for new content in the data source\n",
    "    \"fieldMappings\": [\n",
    "        {\n",
    "          \"sourceFieldName\" : \"metadata_title\",\n",
    "          \"targetFieldName\" : \"title\"\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"metadata_storage_name\",\n",
    "          \"targetFieldName\" : \"name\"\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"metadata_storage_path\",\n",
    "          \"targetFieldName\" : \"location\"\n",
    "        }\n",
    "    ],\n",
    "    \"outputFieldMappings\":[],\n",
    "    \"parameters\":\n",
    "    {\n",
    "        \"maxFailedItems\": -1,\n",
    "        \"maxFailedItemsPerBatch\": -1,\n",
    "        \"configuration\":\n",
    "        {\n",
    "            \"dataToExtract\": \"contentAndMetadata\",\n",
    "            \"imageAction\": \"generateNormalizedImages\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexers/\" + indexer_name,\n",
    "                 data=json.dumps(indexer_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Uncomment if you find an error\n",
    "# r.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: If you get a 400 unauthorize error, make sure that you are using the Azure Search MANAGEMENT KEY, not the QUERY key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n",
      "Status: success\n",
      "Items Processed: 3107\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Optionally, get indexer status to confirm that it's running\n",
    "try:\n",
    "    r = requests.get(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexers/\" + indexer_name +\n",
    "                     \"/status\", headers=headers, params=params)\n",
    "    # pprint(json.dumps(r.json(), indent=1))\n",
    "    print(r.status_code)\n",
    "    print(\"Status:\",r.json().get('lastResult').get('status'))\n",
    "    print(\"Items Processed:\",r.json().get('lastResult').get('itemsProcessed'))\n",
    "    print(r.ok)\n",
    "    \n",
    "except Exception as e:\n",
    "    print(\"Wait a few seconds until the process starts and run this cell again.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**When the indexer finishes running we will have all 994 documents indexed in your Search Engine!.**\n",
    "\n",
    "**Note:** Noticed that it only index 1 document (the zip file) but the AI Search service did the work of uncompressing it and indexing each individual doc**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "- https://learn.microsoft.com/en-us/azure/search/cognitive-search-tutorial-blob\n",
    "- https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/search\n",
    "- https://learn.microsoft.com/en-us/azure/search/search-get-started-vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "In the next notebook 02, we will implement another type of indexing call One-to-Many, in which a single CSV or JSON file can be converted into multiple individual searchable documents in Azure Search. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "GPTSearch3 (Python 3.12)",
   "language": "python",
   "name": "gptsearch3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "9ff083f0c83558f9261023d47a77b9b3eb892c62cdbe066d046abcad1a5edb5c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}