3_rag_evaluation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OSv-zDMdL9R4"
      },
      "source": [
        "# <b>Interactive Exploration of RAG and the RAGAS Framework:</b>\n",
        "\n",
        "---\n",
        "\n",
        "<b><a href=\"https://arxiv.org/abs/2309.15217\">PAPER: RAGAS: Automated Evaluation of Retrieval Augmented Generation</a></b>\n",
        "\n",
        "<br>\n",
        "\n",
        "<center><img src=\"https://github.com/explodinggradients/ragas/raw/main/docs/_static/imgs/logo.png\"></center>\n",
        "\n",
        "---\n",
        "\n",
        "<i>Github Link: <b><a href=\"https://github.com/explodinggradients/ragas\">explodinggradients/ragas</a></b></i>\n",
        "<i>Langchain Blog: <b><a href=\"https://blog.langchain.dev/evaluating-rag-pipelines-with-ragas-langsmith/\">Evaluating RAG Pipelines with Ragas and Langsmith</a></b></i>\n",
        "<br>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mRb_64CxJhfH"
      },
      "source": [
        "<br>\n",
        "\n",
        "## <b>RAG OVERVIEW</b>\n",
        "\n",
        "---\n",
        "\n",
        "<br>\n",
        "\n",
        "### <b>Understanding RAG</b>\n",
        "\n",
        "I suggest to checkout this [link](https://www.promptingguide.ai/techniques/rag) for understanding about Retrieval Augmented Generation.\n",
        "<br>\n",
        "\n",
        "<img src=\"https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9ee6310-47da-4661-958c-a2bdc069c2b7_1464x855.png\" width=100%>\n",
        "\n",
        "<br>\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m_BJi9CYXkv7"
      },
      "source": [
        "<br>\n",
        "\n",
        "## <b>RAGAS RESEARCH PAPER OVERVIEW</b>\n",
        "\n",
        "---\n",
        "\n",
        "### <b>Understanding RAGAS: A Friendly Guide to Evaluating AI's Smart Answers</b>\n",
        "\n",
        "\n",
        "**A few basics regarding RAGAS.**\n",
        "\n",
        "<br>\n",
        "\n",
        "#### **What's RAGAS All About?**\n",
        "RAGAS stands for **Retrieval Augmented Generation Assessment**:\n",
        "\n",
        "- **Retrieval Augmented Generation (RAG)**: This is like pulling out the most accurate answer for a particular question from a pool of data.\n",
        "\n",
        "- **Why RAGAS?** So, how do we know if the AI is really making sense with the data it fetches? That's where RAGAS comes in. It's like a scorecard for our AI's homework. Note that it's not just about accuracy, we can actually see an evaluation of different categories that gives us insight into what parts of RAG went well and what did not. So, RAGAS gives out scores like faithfulness, answer relevancy score, context relevance, context recall score.\n",
        "\n",
        "<br>\n",
        "\n",
        "<img src=\"https://blog.langchain.dev/content/images/size/w1000/2023/08/image-21.png\">\n",
        "\n",
        "<br>\n",
        "\n",
        "\n",
        "RAGAS grades our AI in four key areas:\n",
        "\n",
        "1. **Faithfulness**: Is our AI telling truths? Essentially, are the answers based on the data it found fom the knowledge base, or is it making stuff up?\n",
        "\n",
        "2. **Answer Relevance**: Did the AI really get what you are asking?\n",
        "\n",
        "3. **Context Relevance/Precision**: Did the AI pick the right context? It's no good if it fetches about receipes when one asked about green energy!\n",
        "\n",
        "4. **Context Recall**: Can the AI get all the relevant things to answer a question?\n",
        "\n",
        "<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "75HWjpCqLrtY"
      },
      "source": [
        "<br>\n",
        "\n",
        "## **IMPORTS**\n",
        "\n",
        "\n",
        "For this to work you will need to either have your enviornment file in the same folder where this code lies. This needs to have an api key for openai for anything to work.\n",
        "\n",
        "This file will have a single line **OPENAI_API_KEY=your-openai-key**\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install python-dotenv"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xsi_P9s1Bgq2",
        "outputId": "9ddb451e-0d64-4027-d160-d8ff335a09b4"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting python-dotenv\n",
            "  Downloading python_dotenv-1.0.0-py3-none-any.whl (19 kB)\n",
            "Installing collected packages: python-dotenv\n",
            "Successfully installed python-dotenv-1.0.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "AFM84XpeOprZ"
      },
      "outputs": [],
      "source": [
        "# This is for environment variables...\n",
        "import os\n",
        "OPENAI_API_KEY = os.environ[\"OPENAI_API_KEY\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VcoScx4gIhc_",
        "outputId": "5aa4b649-fdc2-45b9-e41c-8b5fe8fff168"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/2.0 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.2/2.0 MB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.6/2.0 MB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/2.0 MB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━\u001b[0m \u001b[32m1.8/2.0 MB\u001b[0m \u001b[31m13.0 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m12.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (0.1.99)\n",
            "\t\t– MATPLOTLIB VERSION: 3.7.1\n"
          ]
        }
      ],
      "source": [
        "!pip install -q --upgrade duckduckgo-search # for rag\n",
        "!pip install -q --upgrade google-search-results # for rag\n",
        "!pip install -q wikipedia # for rag\n",
        "!pip install -q faiss-cpu\n",
        "!pip install -q unstructured # for website scraping\n",
        "!pip install -q --upgrade langchain\n",
        "!pip install -q --upgrade openai\n",
        "!pip install sentencepiece\n",
        "!pip install -q ragas\n",
        "\n",
        "# Machine Learning Imports (basics)\n",
        "import torch\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import sklearn\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "import scipy\n",
        "\n",
        "pd.options.mode.chained_assignment = None\n",
        "pd.set_option('display.max_columns', None)\n",
        "\n",
        "# For GPU usage in cuda/hf/torch\n",
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "\n",
        "# LLM Utility Imports\n",
        "from dotenv import load_dotenv, find_dotenv\n",
        "\n",
        "# Built-In Imports (mostly don't worry about these)\n",
        "from collections import Counter\n",
        "from typing import List, Union\n",
        "from datetime import datetime\n",
        "from zipfile import ZipFile\n",
        "from glob import glob\n",
        "import warnings\n",
        "import requests\n",
        "import hashlib\n",
        "import imageio\n",
        "import IPython\n",
        "import sklearn\n",
        "import urllib\n",
        "import zipfile\n",
        "import pickle\n",
        "import random\n",
        "import shutil\n",
        "import string\n",
        "import json\n",
        "import math\n",
        "import time\n",
        "import gzip\n",
        "import ast\n",
        "import sys\n",
        "import io\n",
        "import os\n",
        "import gc\n",
        "import re\n",
        "\n",
        "# Visualization Imports (overkill)\n",
        "import matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\n",
        "from PIL import Image, ImageEnhance; Image.MAX_IMAGE_PIXELS = 5_000_000_000;\n",
        "from tqdm.notebook import tqdm; tqdm.pandas();\n",
        "from IPython.display import HTML\n",
        "import matplotlib.pyplot as plt\n",
        "import plotly.express as px\n",
        "import seaborn as sns\n",
        "import plotly\n",
        "import PIL\n",
        "import cv2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "Vh7Ux0UfQU4U"
      },
      "outputs": [],
      "source": [
        "# Langchain Imports\n",
        "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
        "from langchain.vectorstores import FAISS\n",
        "from langchain.embeddings import OpenAIEmbeddings\n",
        "from langchain.tools import DuckDuckGoSearchResults\n",
        "from langchain.tools import DuckDuckGoSearchRun\n",
        "from langchain.tools.base import StructuredTool\n",
        "from langchain.chat_models import ChatOpenAI\n",
        "from langchain.chains import LLMChain\n",
        "from langchain.chains import RetrievalQA\n",
        "from langchain.prompts import BaseChatPromptTemplate\n",
        "from langchain.prompts import PromptTemplate\n",
        "from langchain.document_loaders import UnstructuredURLLoader\n",
        "from langchain.utilities import DuckDuckGoSearchAPIWrapper\n",
        "from langchain.utilities import SerpAPIWrapper\n",
        "from langchain.schema import Document, AgentAction, AgentFinish, HumanMessage\n",
        "from langchain.agents import Tool, AgentExecutor, LLMSingleActionAgent, AgentOutputParser\n",
        "from langchain.retrievers import WikipediaRetriever\n",
        "\n",
        "# Ragas Imports\n",
        "import ragas\n",
        "from ragas.metrics import faithfulness, answer_relevancy, context_relevancy, context_recall\n",
        "from ragas.langchain import RagasEvaluatorChain\n",
        "from transformers import pipeline,AutoTokenizer,AutoModelForSeq2SeqLM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q3gJz0mHW1Zz"
      },
      "source": [
        "## <b>LANGCHAIN BASICS</b>\n",
        "\n",
        "\n",
        "We're using langchain library to instantiate an embedding model (ada2) and a chat model(gpt-3.5-turbo) that can take in a prompt with N number of injectable kwargs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XeCZFeA0SnXg",
        "outputId": "04ce83a8-4714-4bd7-abd4-477da2fff175"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "What word do you want to find similar/related words for?\n",
            "sharp\n",
            "\n",
            "\n",
            "LLM Output for the Prompt: What are the five words that are related to the word sharp. Format the output as a newline seperated list.\n",
            "\n",
            "precise\n",
            "keen\n",
            "pointed\n",
            "acute\n",
            "edged\n",
            "COSINE SIMILARITY BETWEEN INPUT AND OUTPUT VECTORS\n",
            "[[0.84327639 0.85373873 0.85730886 0.85713171 0.83342842]]\n",
            "\n",
            "TOP 3 MOST SIMILAR WORDS TO THE INPUT WORD:\n",
            "[('pointed', 0.8573088581393408), ('acute', 0.8571317097688629), ('keen', 0.8537387321245584)]\n",
            "\n",
            "TOP 3 LEAST SIMILAR WORDS TO THE INPUT WORD:\n",
            "[('keen', 0.8537387321245584), ('precise', 0.8432763915755322), ('edged', 0.8334284175408048)]\n"
          ]
        }
      ],
      "source": [
        "def create_embedding_model(model_name=\"text-embedding-ada-002\"):\n",
        "    embeddings = OpenAIEmbeddings(\n",
        "        model=model_name,\n",
        "    )\n",
        "    return embeddings\n",
        "\n",
        "\n",
        "def create_openai_model(model_name='gpt-3.5-turbo', temperature=0.7, streaming=True, streaming_cb=None):\n",
        "    if streaming_cb:\n",
        "        callbacks=[streaming_cb]\n",
        "    else:\n",
        "        callbacks=[StreamingStdOutCallbackHandler()]\n",
        "\n",
        "    chat = ChatOpenAI(\n",
        "        model=model_name,\n",
        "        temperature=temperature,\n",
        "        streaming=streaming,\n",
        "        callbacks=callbacks if streaming else None\n",
        "    )\n",
        "    return chat\n",
        "\n",
        "chat_model = create_openai_model()\n",
        "embedding_model = create_embedding_model()\n",
        "prompt = \"What are the five words that are related to the word {word}. Format the output as a newline seperated list.\"\n",
        "user_input = input(\"What word do you want to find similar/related words for?\\n\")\n",
        "chain = LLMChain(llm=chat_model, prompt=PromptTemplate.from_template(prompt))\n",
        "\n",
        "print(f\"\\n\\nLLM Output for the Prompt: {prompt.format(word=user_input)}\\n\")\n",
        "output = chain.run(word=user_input)\n",
        "output_words = output.split()\n",
        "output_embeddings = np.array(embedding_model.embed_documents([user_input,]+output_words))\n",
        "input_word_embeddings, output_word_embeddings = output_embeddings[:1, :], output_embeddings[1:, :]\n",
        "print(\"\\nCOSINE SIMILARITY BETWEEN INPUT AND OUTPUT VECTORS\")\n",
        "similarities = cosine_similarity(input_word_embeddings, output_word_embeddings)\n",
        "print(similarities)\n",
        "\n",
        "print(\"\\nTOP 3 MOST SIMILAR WORDS TO THE INPUT WORD:\")\n",
        "words_sorted_by_cos_sim = sorted(zip(output_words, similarities[0]), key=lambda x: x[1], reverse=True)  # Sort by float value in descending order\n",
        "print(words_sorted_by_cos_sim[:3])\n",
        "\n",
        "print(\"\\nTOP 3 LEAST SIMILAR WORDS TO THE INPUT WORD:\")\n",
        "print(words_sorted_by_cos_sim[-3:])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQQaZoJSZDQb"
      },
      "source": [
        "<br>\n",
        "\n",
        "## <b>DATA SOURCE(S)</b>\n",
        "\n",
        "---\n",
        "\n",
        "For our experiments I will be using DuckDuckGo search to return a number of webpages related to the user's search which we will then scrape and split. We will also utilize wikipedia search.\n",
        "\n",
        "We will create a small FAISS vectorstore and use that for RAG."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SyafLyf2SP5M"
      },
      "source": [
        "<br>\n",
        "\n",
        "## <b>THE RETRIEVER</b>\n",
        "\n",
        "---\n",
        "\n",
        "It retrieves and embeds web and Wikipedia documents related to a given query into a FAISS vector store.\n",
        "\n",
        "This function performs several steps to gather and process information related to a given query:\n",
        "\n",
        "1. Executes a DuckDuckGo search for the query and retrieves corresponding URLs.\n",
        "2. Splits the webpage contents into Langchain documents.\n",
        "3. Fetches relevant Wikipedia documents based on the query.\n",
        "4. Embeds all gathered documents into a FAISS vector store for further processing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "U6l1Ze-daF4-"
      },
      "outputs": [],
      "source": [
        "def query_to_vector(query, ddg_wrapper=None, wiki_retriever=None, embedding_model=None, n_results=4):\n",
        "    if wiki_retriever is None:\n",
        "        wiki_retriever = WikipediaRetriever()\n",
        "\n",
        "    if ddg_wrapper is None:\n",
        "        ddg_wrapper = DuckDuckGoSearchAPIWrapper(max_results=n_results)\n",
        "\n",
        "    if embedding_model is None:\n",
        "        embedding_model = create_embedding_model()\n",
        "\n",
        "    docs = UnstructuredURLLoader(\n",
        "        [x['link'] for x in ddg_wrapper.results(query, num_results=n_results)],\n",
        "        show_progress_bar=True, headers={\"User-Agent\": \"value\"}\n",
        "    ).load_and_split()\n",
        "    docs = [x for x in docs if len(x.page_content)>100]\n",
        "\n",
        "    try:\n",
        "        docs += wiki_retriever.get_relevant_documents(query=query)\n",
        "    except:\n",
        "        print(\"\\n[WARNING] retrieving from wikipedia failed [/WARNING]\\n\")\n",
        "        pass\n",
        "\n",
        "    vector_ = FAISS.from_documents(docs, embedding_model)\n",
        "    return vector_"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7tb3NdNpSOiR"
      },
      "source": [
        "<br>\n",
        "\n",
        "## <b>GENERATION</b>\n",
        "For this we use the RetrievalQA chain by langchain."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "21wHErGcTHG1",
        "outputId": "a44c73ba-57a5-43d6-e37b-b4afce6f6a86"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "What do you want to ask?\n",
            ">>> What are the major impacts on VMware after Broadcom VMware's acquisition?\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r  0%|          | 0/4 [00:00<?, ?it/s][nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "100%|██████████| 4/4 [00:07<00:00,  1.86s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The major impacts on VMware after Broadcom's acquisition include layoffs and redundancies. Over 2,800 VMware employees have been laid off or made redundant by Broadcom post-acquisition. There have also been changes to the senior management team, with the former CEO, Raghu Raghuram, stepping down and several other senior executives leaving the company. Additionally, there have been reports of several VMware offices being closed down globally. It is unclear at this time how these changes will affect VMware's operations and future direction under Broadcom's ownership.{'query': \"What are the major impacts on VMware after Broadcom VMware's acquisition?\", 'result': \"The major impacts on VMware after Broadcom's acquisition include layoffs and redundancies. Over 2,800 VMware employees have been laid off or made redundant by Broadcom post-acquisition. There have also been changes to the senior management team, with the former CEO, Raghu Raghuram, stepping down and several other senior executives leaving the company. Additionally, there have been reports of several VMware offices being closed down globally. It is unclear at this time how these changes will affect VMware's operations and future direction under Broadcom's ownership.\", 'source_documents': [Document(page_content='acquired and fired    —\\n\\nBroadcom cuts at least 2,800 VMware jobs following $69 billion acquisition\\n\\nBroadcom hasn\\'t said how many people will be affected, or much of anything else.\\n\\nAndrew Cunningham\\n    -    Dec 1, 2023 8:16 pm UTC\\n\\nEnlarge\\n\\nVMWare\\n\\nreader comments\\n  \\n  144\\n\\nBroadcom announced back in May of 2022 that it would buy VMware for $61 billion and take on an additional $8 billion of the company\\'s debt, and on November 22 of 2023 Broadcom said that it had completed the acquisition. And it looks like Broadcom\\'s first big move is going to be layoffs: according to WARN notices filed with multiple states (catalogued here by Channel Futures), Broadcom will be laying off at least 2,837 employees across multiple states, including 1,267 at its Palo Alto campus in California.\\n\\nAs Channel Futures notes, the actual number of layoffs could be higher, since not all layoffs require WARN notices. We\\'ve contacted Broadcom for more information about the total number of layoffs and the kinds of positions that are being affected and will update if we receive a response. VMware has around 38,300 employees worldwide.\\n\\nAdvertisement\\n\\nThe WARN notices list the reason for the layoffs as \"economic,\" but provide no further explanation or justification.\\n\\nFurther Reading\\n\\nBroadcom plans a “rapid transition” to subscription revenue for VMware\\n\\nBroadcom still makes most of its money from its wired and wireless communication products and other chips; 78 percent of the roughly $8.9 billion that Broadcom made in Q3 of 2023 was from its \"semiconductor solutions\" segment. But Broadcom has been trying to buy its way into enterprise software for years now, including an $18.9 billion acquisition of CA Technologies in 2018 and a $10.7 billion acquisition of Symantec in 2019.\\n\\nDespite the layoffs, VMware will remain central to Broadcom\\'s enterprise software strategy. The company\\'s Broadcom Software Group will be renamed VMware, and the company is reportedly planning to emphasize software subscriptions rather than one-time sales to boost revenue.\\xa0Aside from that, we don\\'t know many details about what a Broadcom-owned VMware will ultimately look like. The only VMware product mentioned at any length in Broadcom CEO Hock Tan\\'s post about the acquisition is VMware Cloud Foundation.\\n\\nPrior to being owned by Broadcom, VMware had been part of Dell, a company better known for its consumer and business PCs, servers, and workstations. Dell sold off all of its shares of VMware in 2021, though Michael Dell himself remained the chairman of VMware\\'s board.\\n\\nreader comments\\n  \\n  144\\n\\nAndrew Cunningham\\n        Andrew is a Senior Technology Reporter at Ars Technica, with a focus on consumer tech including computer hardware and in-depth reviews of operating systems like Windows and macOS. Andrew lives in Philadelphia and co-hosts a weekly book podcast called\\n\\nOverdue.\\n\\nAdvertisement\\n\\nChannel Ars Technica\\n\\n← Previous story\\n\\nNext story →\\n\\nRelated Stories\\n\\nToday on Ars', metadata={'source': 'https://arstechnica.com/information-technology/2023/12/broadcom-cuts-at-least-2800-vmware-jobs-following-69-billion-acquisition/'}), Document(page_content='15\\nSOCIAL BUZZ\\n\\nFormer VMware CEO speaking at VMware Explore.\\n\\nVMware by Broadcom: layoffs and redundancy\\n\\nBy Aaron Raj | 5 December, 2023\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nOver 2800 VMware employees have been laid off or made redundant by Broadcom post-acquisition.\\n\\nBroadcom CEO insists company will remained focus on growing VMware.\\n\\nCEO also requested VMware employees to return to office to work.\\n\\nAs soon as Broadcom announced its plans to acquire VMware, many in the industry understood that layoffs would be in store for staff should the deal go through. And true enough, since the deal closed, more than two thousand VMware employees have since been laid off or made redundant by Broadcom.\\n\\nVMware has over 38,000 employees worldwide. In the US, a report by ChannelFutures stated that data stems from WARN notices the acquisitive chipmaker has filed in multiple states. The Worker Adjustment and Retraining Notification (WARN)\\xa0Act\\xa0helps ensure advance notice in cases of qualified plant closings and mass layoffs.\\n\\nVMWARE KEEPS MUM ON LAYOFFS AS BROADCOM ACQUISITION CLOSESAaron Raj | 30 October, 2023\\n\\nBroadcom’s actions post-acquisition are nothing new. The company made similar moves when it acquired CA Technologies, Symantec and several others in the past. But most of the companies it acquired previously were neither as profitable or innovative as VMware.\\n\\n“While an important moment for Broadcom, it’s also an exciting milestone for our customers around the world. And as I said when we first announced the acquisition, we can now come together and have the scale to help global enterprises address their complex IT infrastructure challenges by enabling private and hybrid cloud environments and helping them deploy an “apps anywhere” strategy. Our goal is to help customers optimize their private, hybrid and multi-cloud environments, allowing them to run applications and services anywhere,” said Hock Tan, president and CEO of Broadcom in a statement.\\n\\nWhile Tan has hinted that the company will remain committed to several offerings by VMware, there have been questions asked about the areas that were not mentioned. That includes VMware’s cybersecurity venture – Carbon Black.\\n\\nCarbon Black may no longer be a key business unit for VMware post-acquisition. Broadcom already has several cybersecurity ventures under its group and a recent report from CRN stated that Broadcom had told VMware channel partners to immediately stop bundling sales of VMware products with those belonging to Carbon Black.\\n\\nThis is the first of many changes implemented by Broadcom. In an email and town hall meeting with VMware employees, Tan said he plans to review strategic alternatives for both Carbon Black and VMware’s end-user computing practice.\\n\\nBroadcom CEO speaking to VMware employees. (Source – LinkedIn)\\n\\nNew leaders for VMware?\\n\\nWith layoffs ongoing at VMware, there have also been some changes made to the senior management team of the company. Raghu Raghuram is now the former CEO of VMware. The man who remained optimistic about the deal, posted on LinkedIn that he will now be taking a break and will remain to serve as a strategic advisor to Tan.\\n\\n“During this time, we have navigated a dizzying journey together that spans the full lifecycle of a business – from raising debt financing to spinning off from Dell and operating as a standalone public company, to being acquired by Broadcom and executing through a complex regulatory process to finally arrive at this milestone moment in VMware’s history. Throughout it all, I have been amazed at the resiliency and focus of the VMware team as we repeatedly beat our financial goals, broke new ground with our product releases and took time to celebrate the values that make us a special company,” said Raghuram in his post.\\n\\nNUTANIX CONFIDENT ON BOOSTING CUSTOMERS AS BROADCOM CLOSES VMWARE ACQUISITION\\xa0Aaron Raj | 23 November, 2023', metadata={'source': 'https://techwireasia.com/2023/12/what-happens-next-with-vmware-after-layoffs-and-redundancy/'}), Document(page_content='NUTANIX CONFIDENT ON BOOSTING CUSTOMERS AS BROADCOM CLOSES VMWARE ACQUISITION\\xa0Aaron Raj | 23 November, 2023\\n\\nAlso leaving the company is Sumit Dhawan. The former president of VMware has now joined Proofpoint as its new CEO. In his most recent role as president of VMware, Dhawan was responsible for driving over US$13B of revenue and led the company’s go-to-market functions including worldwide sales, customer success and experience, strategic ecosystem, industry solutions, marketing, and communications.\\n\\nApart from Dhawan and Raghuram, several other senior executives have also announced plans to move away from VMware. But that’s not all. There have also been reports of several VMware offices being closed down as well around the world.\\n\\nIn the US, reports state that some of Broadcom’s employees will also now move to VMware’s campus office in Palo Alto. In an interview, Tan also requested VMware employees living within 60 miles of a Broadcom location to return to the office to work.\\n\\n“If you are customer-facing, go-to-market (sales), then sure, you can be remote, I don’t care…Any other exception, you better learn how to walk on water. I’m serious,” Tan said in an interview.\\n\\nThere could be more layoffs at VMware in the months to come.\\n\\nVMware layoffs and redundancy in Southeast Asia\\n\\nThe layoffs and redundancy announced by Broadcom seem to have a major impact on VMware’s Southeast Asia offices. In Malaysia, the entire VMware office has been shut down, while several channel sales partners in Singapore have also been made redundant following the acquisition.\\n\\nA conversation with a former executive of VMware revealed that many employees in the regional headquarters are still unsure if they would want to remain with the company. He said that many were not pleased with the way Broadcom deals with clients and partners, and that he could see clients moving away from relying on VMware products in the future.\\n\\nHe also said that Broadcom is quite adamant in how it conducts its businesses. While the company is profitable and doing things right, its approach to sales may be different from how VMware operated in the past. And this is where VMware employees might find it more challenging to cope with – and could be made redundant.\\n\\nAs such, VMware employees may have a bleak future on the horizon, but it doesn’t mean that it’s the endgame for them or the company. While many will undoubtedly leave in search of better opportunities, there will also be those who remain and continue to support Broadcom in its plans to grow VMware’s business.\\n\\nAfter all, Tan has acknowledged that Broadcom qill continue to invest in VMware’s innovations and continue to support the development of the company’s products. The only question will be whether the team that remains will be open to Broadcom’s terms of doing business or will decide to move on.\\n\\nBROADCOM\\n\\nTECH LAYOFFS\\n\\nVMWARE\\n\\n15\\nSOCIAL BUZZ\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nBy Aaron Raj\\n\\nAaron Raj\\n\\nAaron enjoys writing about enterprise technology in the region. He has attended and covered many local and international tech expos, events and forums, speaking to some of the biggest tech personalities in the industry. With over a decade of experience in the media, Aaron previously worked on politics, business, sports and entertainment news.\\n\\nREAD MORE\\n\\nCyber-heist mastery: how North Korea stole over US$3 billion in cryptocurrency\\n\\nFrom 1% to 100%: Tallying the impact from Okta data breach\\n\\nChatGPT: A year of revolutionizing AI dynamics\\n\\nBarking up the wrong data tree: even pets aren’t safe from a data breach\\n\\nChipping away: Nvidia CEO predicts a long journey for US chip independence\\n\\nTRENDING TOPICS\\n\\nCYBERATTACKS\\n\\nDATA BREACH\\n\\nBROADCOM\\n\\nGENERATIVE AI\\n\\nCYBER THREATS\\n\\nSUPPLY CHAIN\\n\\nAI\\n\\nLOGISTICS\\n\\nARTIFICIAL INTELLIGENCE\\n\\nDATA CENTER', metadata={'source': 'https://techwireasia.com/2023/12/what-happens-next-with-vmware-after-layoffs-and-redundancy/'}), Document(page_content='News Releases\\n\\n4 min read\\n\\nBroadcom and VMware Intend to Close Transaction on November 22, 2023\\n\\nShare \"Broadcom and VMware Intend to Close Transaction on November 22, 2023\" on Twitter\\n\\nShare \"Broadcom and VMware Intend to Close Transaction on November 22, 2023\" on Facebook\\n\\nShare \"Broadcom and VMware Intend to Close Transaction on November 22, 2023\" on LinkedIn\\n\\nReceives Final Regulatory Approval for Transaction\\n\\nSAN JOSE, Calif. and PALO ALTO, Calif., November 21, 2023 – Broadcom Inc. (NASDAQ: AVGO) and VMware, Inc. (NYSE: VMW) today announced that they have received all required regulatory approvals and intend to close Broadcom’s acquisition of VMware on November 22, 2023.\\n\\nBroadcom has received legal merger clearance in Australia, Brazil, Canada, China, the European Union, Israel, Japan, South Africa, South Korea, Taiwan, the United Kingdom, and foreign investment control clearance in all necessary jurisdictions. There is no legal impediment to closing under U.S. merger regulations.\\n\\nAbout Broadcom\\n\\nBroadcom Inc. (NASDAQ: AVGO), a Delaware corporation headquartered in San Jose, CA, is a global technology leader that designs, develops and supplies a broad range of semiconductor and infrastructure software solutions. Broadcom\\'s category-leading product portfolio serves critical markets including data center, networking, enterprise software, broadband, wireless, storage and industrial. Our solutions include data center networking and storage, enterprise, mainframe and cybersecurity software focused on automation, monitoring and security, smartphone components, telecoms and factory automation.\\n\\nAbout VMware\\n\\nVMware is a leading provider of multi-cloud services for all apps, enabling digital innovation with enterprise control. As a trusted foundation to accelerate innovation, VMware software gives businesses the flexibility and choice they need to build the future. Headquartered in Palo Alto, California, VMware is committed to building a better future through the company\\'s 2030 Agenda. For more information, please visit www.VMware.com/company.\\n\\nCautionary Statement Regarding Forward-Looking Statements\\n\\nThis communication relates to a proposed business combination transaction between Broadcom and VMware. This communication includes forward-looking statements within the meaning of Section 21E of the U.S. Securities Exchange Act of 1934, as amended, and Section 27A of the U.S. Securities Act of 1933, as amended. These forward-looking statements include but are not limited to statements that relate to the anticipated closing date of the proposed transaction. These forward-looking statements are identified by words such as \"will,\" \"expect,\" \"believe,\" \"anticipate,\" \"estimate,\" \"should,\" \"intend,\" \"plan,\" \"potential,\" \"predict,\" \"project,\" \"aim,\" and similar words or phrases. These forward-looking statements are based on current expectations and beliefs of Broadcom and VMware management and current market trends and conditions.', metadata={'source': 'https://news.vmware.com/releases/broadcom-and-vmware-intend-to-close-transaction-on-november-22-2023'})]}\n"
          ]
        }
      ],
      "source": [
        "# Create the llm (feel free to change the temp)\n",
        "TEMP = 0.5\n",
        "llm = create_openai_model(temperature=TEMP)\n",
        "\n",
        "# Pick a question\n",
        "user_provided_question = input(\"What do you want to ask?\\n>>> \")\n",
        "\n",
        "# Instantiate the chain and query it\n",
        "qa_chain = RetrievalQA.from_chain_type(\n",
        "    llm,\n",
        "    retriever=query_to_vector(user_provided_question).as_retriever(),\n",
        "    return_source_documents=True\n",
        ")\n",
        "result = qa_chain({\"query\": user_provided_question})\n",
        "\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bGPosKl0UPRH"
      },
      "source": [
        "<br>\n",
        "\n",
        "## <b>EVALUATION</b>\n",
        "\n",
        "---\n",
        "\n",
        "For this we will use the 4 eval metrics provided by RAGAS:\n",
        "1. Faithfulness\n",
        "2. Answer Relevancy\n",
        "3. Context Relevancy (precision)\n",
        "4. Context Recall\n",
        "\n",
        "NOTE: If we want to use certain metrics, we also need to provide the ground truth answer when building the QA chain. **`ground_truths`** in the `qa_chain` call\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "nfFf3jffnkML",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0310bc76-9c16-45d0-ba02-9810e370c124"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 4/4 [00:06<00:00,  1.66s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Full fine-tuning and PEFT (Parameter-efficient Fine-tuning) are two different approaches to fine-tuning a pretrained language model. Here's how they differ:\n",
            "\n",
            "1. Scope of Parameter Updates:\n",
            "- Full Fine-tuning: In full fine-tuning, all the parameters of the pretrained language model are updated during the fine-tuning process. This means that the entire model is adjusted to optimize its performance for a specific task or set of tasks. It is similar to the pretraining process, but done on a smaller, task-specific dataset.\n",
            "- PEFT: In PEFT, only a small number of parameters in the pretrained model are updated. Instead of adjusting the entire model, PEFT focuses on selectively modifying specific components or adding smaller additional components (e.g., adapter layers) to the model. This approach aims to achieve efficient adaptation by leveraging the pretrained model's existing knowledge and only updating the necessary parameters.\n",
            "\n",
            "2. Computational Cost:\n",
            "- Full Fine-tuning: Full fine-tuning can be computationally expensive, especially for large language models with tens or hundreds of billions of parameters. It requires significant computing power and memory resources to train the model on a task-specific dataset.\n",
            "- PEFT: PEFT is designed to be more computationally efficient compared to full fine-tuning. By updating only a small number of parameters, PEFT reduces the computational cost and memory requirements of the fine-tuning process. This makes it a more cost-effective option, especially for businesses with limited computing resources.\n",
            "\n",
            "3. Expertise and Implementation Complexity:\n",
            "- Full Fine-tuning: Implementing full fine-tuning requires a deep understanding of deep learning, natural language processing (NLP), and data science. It involves expertise in adjusting the model's weights through tuning scripts, selecting appropriate hyperparameters, and managing the training process. It is a complex process that demands ML expertise.\n",
            "- PEFT: PEFT is generally considered to be less complex to implement compared to full fine-tuning. While it still requires knowledge of deep learning and NLP, it focuses on specific techniques (e.g., low-rank approximation) to update a subset of parameters. PEFT can be more accessible to practitioners with intermediate ML knowledge and can be a good option for efficient adaptation.\n",
            "\n",
            "In summary, full fine-tuning updates all parameters of the pretrained model, while PEFT selectively updates a small number of parameters. PEFT is designed to be computationally efficient and requires less expertise compared to full fine-tuning. The choice between the two methods depends on factors such as computational resources, expertise, and the specific requirements of the task at hand.\n",
            "\n",
            "RAGAS EVALUATION\n",
            "\tfaithfulness_score: 0.6666666666666667\n",
            "\tanswer_relevancy_score: 0.9318176979166825\n",
            "\tcontext_relevancy_score: 0.00784313725490196\n",
            "\tcontext_recall_score: 1.0\n"
          ]
        }
      ],
      "source": [
        "# testing it out\n",
        "question = \"Explain difference between full fine tuning and peft based fine tuning.\"\n",
        "\n",
        "# From google search\n",
        "answer = \"\"\"Full Fine-tuning: Adjusts all parameters of the LLM using task-specific data. Parameter-efficient Fine-tuning (PEFT): Modifies select parameters for more efficient adaptation.\"\"\"\n",
        "\n",
        "# make eval chains\n",
        "eval_chains = {\n",
        "    m.name: RagasEvaluatorChain(metric=m)\n",
        "    for m in [faithfulness, answer_relevancy, context_relevancy, context_recall]\n",
        "}\n",
        "llm = create_openai_model(temperature=0.5)\n",
        "qa_chain = RetrievalQA.from_chain_type(\n",
        "    llm, retriever=query_to_vector(question).as_retriever(),\n",
        "    return_source_documents=True\n",
        ")\n",
        "result = qa_chain({\"query\": question, \"ground_truths\":answer})\n",
        "model_result = result[\"result\"]\n",
        "\n",
        "\n",
        "# evaluate\n",
        "print(\"\\n\\nRAGAS EVALUATION\")\n",
        "for name, eval_chain in eval_chains.items():\n",
        "    score_name = f\"{name}_score\"\n",
        "    print(f\"\\t{score_name}: {eval_chain(result)[score_name]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## <b>Conclusion</b>\n",
        "\n",
        "Here, we have explored how to build and evaluate a RAG pipeline using LLMChains, with a specific focus on evaluating the generated responses within the pipeline."
      ],
      "metadata": {
        "id": "3Mly95MiqweO"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}