3_Keyword_extraction_model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Keyword_extraction_model.ipynb",
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMD4cTlC3cIacj63zxfP/MY",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dlskawns/cp1/blob/main/3_Keyword_extraction_model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3. 키워드 추출 모델"
      ],
      "metadata": {
        "id": "x3Fqj7ZyvnIh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Sentence Transformer - Bert(distilbert-base-nli-mean-tokens)를 이용 \n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "MpAHzqXUCJbs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### * 선정 이유:  \n",
        "NLI(Natural Language Inference) 작업에 적합한 모델로, 문장(doc)단위 임베딩이 가능하다. \n",
        "\n"
      ],
      "metadata": {
        "id": "_nGQt7bic2BV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### * 가설:  \n",
        "문장단위 임베딩을 진행 후, 단어 임베딩의 유사도를 파악해 해당 문장 내 중요 키워드를 찾아낼 수 있다.\n"
      ],
      "metadata": {
        "id": "zKTws9I4c3se"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### * 진행 방법:  \n",
        "  * 빈도 수 기반 벡터화를 진행 (Counter Vectorizer) - n gram(3,3)으로 단어 묶음 설정\n",
        "  * sentence transformer로 샘플 별 문장 임베딩 생성\n",
        "  * sentence transformer로 샘플 별 단어 임베딩 생성\n",
        "  * Cosine 유사도를 통해 특정 중요 단어 파악\n",
        "  * 동일한 방법으로 n gram(1,1)로 진행하여 최종 키워드 top 5~8개 추출"
      ],
      "metadata": {
        "id": "hnZIb90xc5I0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Model 인스턴스 생성"
      ],
      "metadata": {
        "id": "SXpnc0J2dETB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 트랜스포머 모델 생성\n",
        "model = SentenceTransformer('distilbert-base-nli-mean-tokens')"
      ],
      "metadata": {
        "id": "xtwz00M8AiVo",
        "outputId": "14f9345e-d01f-4147-b4ba-29534e25a8dc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 401,
          "referenced_widgets": [
            "6f17178228af4309801f9feaf2c02345",
            "f0f3429ba5b94c2da4f11cfcbbfd32cf",
            "852b9f73426e4f37a0156e8a439d0724",
            "24edae809dc74dfa90a41a2a024802d0",
            "f7ace139c33a42a58a2ec72acc186888",
            "3bc332fa4b364f79ac66451a3258b101",
            "177f6ac6e5cc468994ca2b1e1d221d8f",
            "20ebeb4cc90443e893d93d9d474a149a",
            "2acb0974869547cc9cf59acca6aded66",
            "207991b2cad94677a890a5797c281e59",
            "723993809c54476f86d20ca34eb139f9",
            "98d2450c6e454b39bab35c0c164bc795",
            "8e11fc30f5844d64a594f2052f4aedc9",
            "1b7c8fdcdb2c4323919277e0db5bab0b",
            "1e405032f802499e9eaab67863133073",
            "fe1d69e698b84ca0a68792602b8f30df",
            "b4deb9aa34c04968a3c865b42c611047",
            "5cc2e773c5434b98829782ec1cd056bb",
            "25f375f2b10d4835908d38efb2c5226c",
            "83469829bc554493a44533fbc873a593",
            "10002aeb681c48d78a56801ceb398af8",
            "1b8442146d8d4b3d91eb508d33d8af5f",
            "0d994fb85adc4dcc8a5758a350092f84",
            "cb662f6635814291b77f5c403a1f6055",
            "4d5ff4e8293e46659663e211d13bfb06",
            "17de965e20494904b31cf920e21dc9c7",
            "7879880d00e94fa68c2c9aeaedbdd363",
            "7cec86c72d3d40b0959814f82ba5f852",
            "2b4a38736d6a4a4d91baf64adc477562",
            "220b8eb4a85548deaf50da8dfd022d38",
            "95b1bd8d0133494a884eb944b9e5b8fb",
            "c749922ac1854dbd8f58432794c8255e",
            "1c19a0ec52974f57b14d4252e6cbc2d5",
            "aa1ed1f04f07408f8626be3c3ea5cbd1",
            "bce954f176484e30937cfe1132c34bf2",
            "72078213a1034cdbbaed97d412925cdb",
            "fb355ae472524c0d9bbcb049850eb6f6",
            "1d67e432a46242f4be64b51eb40a0973",
            "090ae38136794c22920edb6fed85c78f",
            "b7018f92c065403689dadf012c76557c",
            "7d9915eaa46347849e3d7044b7f20c07",
            "2fdfe085d7e44c5780f8772bce8bf6e2",
            "c9bcb30c6b0f40d1b58d428b2b01b8b9",
            "58a24054d44c4d659d077658d08b5f95",
            "6384dd12b4e84f86af8c1ccd56a7a85f",
            "f634312b685f4885b66cc40131b2c448",
            "e0f5339ed0814313b6301045b809f9f1",
            "12b7baf5bd8d419b89f6385281c222f0",
            "91b5ae95bcc74ee08fe24dbe919558f3",
            "cde2ffe48c954f23b8ae8a8a686ba5cb",
            "0fcad1b756ce47dda51a5d5275d44aeb",
            "5bcd0aa4eb94497ab12c43563db85984",
            "e5ca862a21c3473c83d7ae598bad1ca1",
            "c6c7850ce9654f5b8e7e797902953275",
            "f48a5ab0b2194b39bcad77e6c4a7537e",
            "8304dd04f1fe4d0386ad8e78cfb7afe8",
            "1d26ee68c4124130a2afc0b8611f5c5e",
            "71a4a887094044cc95b7bdc3a57ac573",
            "7fd6d06b0abb4434863480f843fd80e8",
            "5ebaaffd9a564f80bff26cbfa7897999",
            "7de855b5087246eca67b810c1f6aedc6",
            "85ef33f2000a4fd881310acf65c10c73",
            "f8a86e45edc841c89861c5c7cddbe77a",
            "dac4021a3d014cd1a3d746780b2318bf",
            "22deb5459839478cbe84a73aa89e576f",
            "9bc1e23366a74999993f6fb15f495e9c",
            "5efc28270db14f5c9ed21c1e0c852530",
            "13d2d1c0a66541e6a966ebbaf14e3660",
            "cb082c76a9344a88b0ab3ac394c733f8",
            "e310ab3e5aef411d85b62de8e27ed71b",
            "7a3f0f687e0046e6af3eab668db0fab5",
            "864d2800fb614310b46c38a8c5a456a6",
            "1d63c6a747704a148d16c4df90a799b8",
            "78e13bba81c1424798469158cf8eb052",
            "86a9466c6d054a7abaf878abd2d40dd2",
            "b67e83efd3474b48b2fe573898a16238",
            "f3b12caa5b8c446fa0cdb7b4d0826673",
            "aa34548a2cc444d09c5e70850054b58d",
            "1ae7d94d9f504b3283f5d690f6ecb908",
            "e2d31a5a49774f0b923a655ff2d9945c",
            "f57fbe5c77ef47498eeeff62d7b4730e",
            "1acc3b1364e641028220214d3821f9c3",
            "b02bc199f9c342efa5ec58680822beb0",
            "a8b6e22e32994f1583648b7b9aaee570",
            "80d4d2e715494174be0a67632f202966",
            "2b7166061a1348a7b35f7b6a91f26a37",
            "6635b2948d2e41edaa79dd6133811bfa",
            "f84116fbd99d4da1b3019189b644cae3",
            "559c091bc8174747b6d8270a33ea4b5d",
            "e06d6edbaf194bd39fb7f0a413529bc1",
            "02aa78275c384095a464b8e58f0a2699",
            "0390ca708f894c8eb157f38758e331b6",
            "b01f979f75ab4977b8662e3541c93c07",
            "81d5a1bcb09b49c1a968780bb4a875be",
            "82e8c37bca5d4017aed24ff18882df29",
            "9bd3ff53b74f4d2c9209ad3c057776e9",
            "44c5ea3c7ab8456ba3667618cccf1726",
            "3809871f82014289ade90eafa5bcdf4a",
            "c0d66a76cca746e0bce40e10e2ae429a",
            "92b7dab392d14620a4387d5968e2499b",
            "9c25411bfe4e4ff5b4d802ad5acff377",
            "c54a0a1dd96b43f787c246dbbdcaba19",
            "0b2acaabde5d4124a06ccd1bd5e5aa8a",
            "2afe18bc9f284c2581078acda4cb8115",
            "438b36306bcc429ca00b84140fbf3a10",
            "847ba416bd254376a11aaf4e3c65e47e",
            "3ea354d01ca84e20b76e634e4d491176",
            "a4f3b6b301b94c1aaef70c4c15a13075",
            "af3ab991d1a34fa6bdebc3307df4ccff",
            "75b2fd0cba8e4dcca5e38074ceea02d0",
            "39e805317cd34501b0c38c01569df694",
            "20f68559ff3c482d98354da4ede44617",
            "942b876103af44d4b6f2f52ac1a7f1cf",
            "e6df7737ee22476d96451fd7769079bb",
            "b4076cafbb5248578d86d5e4d5a6e87a",
            "bfb782b06a4644079d0790e99eef5cb9",
            "8a612cf298ef4b6ca31f7711b9a0f60d",
            "e98ee61cfd3240ce89c20b06a85ecf96",
            "a2fe22de414843889c55d5c07e7a4d5e",
            "e415cd49729c4299b3eca1d5b3a5cafd",
            "c27bb2faf6374e1a8db9512b441194b8",
            "2bfc271336aa40ab8ed314fb6cb34fe7",
            "143ad6c05705404199fbc348bb0b638f",
            "2aa40d55c17844ab9167a726a6140a60",
            "16ba4e82fb9d4d1290e08f43616df1b3",
            "5932295f05e84c31b75edb93d4032e38",
            "5436490eb57440dcab8b6f4859dfe515",
            "197abe02e05b44a28a5617347dae499b",
            "d12dc7c59b354fd7a237cf56c78e7dc5",
            "e0be90af99b947d58ad2cd765d7d1b7d",
            "024645f3871c46c291b6cc9cc9534040",
            "218853d069414108800a9741fe3f94f8"
          ]
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6f17178228af4309801f9feaf2c02345",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/690 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "98d2450c6e454b39bab35c0c164bc795",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/3.99k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "0d994fb85adc4dcc8a5758a350092f84",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/550 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "aa1ed1f04f07408f8626be3c3ea5cbd1",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/122 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6384dd12b4e84f86af8c1ccd56a7a85f",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/229 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8304dd04f1fe4d0386ad8e78cfb7afe8",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/265M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5efc28270db14f5c9ed21c1e0c852530",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/53.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "aa34548a2cc444d09c5e70850054b58d",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/112 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "559c091bc8174747b6d8270a33ea4b5d",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "92b7dab392d14620a4387d5968e2499b",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/450 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "39e805317cd34501b0c38c01569df694",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2bfc271336aa40ab8ed314fb6cb34fe7",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/190 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### 추출모델 작성"
      ],
      "metadata": {
        "id": "W7GeFPtz-rvS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "top_n = 10            # 빈도 수 높은 단어 상위 10개 추출\n",
        "k_word, emb =[], []\n",
        "\n",
        "for i in range(len(df['Text'])):\n",
        "  print(i)\n",
        "  n_gram_range = (3, 3)\n",
        "  # Extract candidate words/phrases\n",
        "  try:  \n",
        "      # 빈도수 기반 벡터화\n",
        "      count = CountVectorizer(ngram_range=n_gram_range, stop_words=stop_words).fit([df['Text'][i]])\n",
        "      candidates = count.get_feature_names()\n",
        "\n",
        "      # 문장 임베딩 \n",
        "      doc_embedding = model.encode([df['Text'][i]])\n",
        "\n",
        "      # 후보 단어 임베딩\n",
        "      candidate_embeddings = model.encode(candidates)\n",
        "\n",
        "      # cosine 유사도 체크\n",
        "      distances = cosine_similarity(doc_embedding, candidate_embeddings)\n",
        "\n",
        "      # 유사도 높은 키워드 추출\n",
        "      keywords = [candidates[index] for index in distances.argsort()[0][-top_n:]]\n",
        "      \n",
        "      # 위의 과정 n gram 줄여서 재진행 \n",
        "      n_gram_range1 = (1, 1)\n",
        "      count1 = CountVectorizer(ngram_range=n_gram_range1, stop_words=stop_words).fit(keywords)\n",
        "      candidates1 = count1.get_feature_names()\n",
        "\n",
        "      doc_embedding1 = model.encode(keywords)\n",
        "      candidate_embeddings1 = model.encode(candidates1)\n",
        "      distances1 = cosine_similarity(doc_embedding1, candidate_embeddings1)\n",
        "\n",
        "      # 명사와 형용사만 추출\n",
        "      a = [candidates1[index] for index in distances1.argsort()[0][-5:] if pos_tag(candidates1[index])[0][1] == 'JJ' if pos_tag(candidates1[index])[0][1] == 'NN' or pos_tag(candidates1[index])[0][1] == 'NNS']\n",
        "      b = [index for index in distances1.argsort()[0][-5:] if pos_tag(candidates1[index])[0][1] == 'JJ' if pos_tag(candidates1[index])[0][1] == 'NN' or pos_tag(candidates1[index])[0][1] == 'NNS']\n",
        "      \n",
        "      # 토큰 자체가가 3개 아래일 경우, 중요단어가 아닌것도 포함\n",
        "      if len(a) < 3:\n",
        "        a = [candidates1[index] for index in distances1.argsort()[0][-5:]]\n",
        "        \n",
        "      k_word.append(a)\n",
        "      # emb.append(b) # 임베딩된 값\n",
        "  except ValueError:\n",
        "    try:\n",
        "      count = CountVectorizer(ngram_range=n_gram_range, stop_words='english').fit([df['Text'][i]])\n",
        "      candidates = count.get_feature_names()\n",
        "      doc_embedding = model.encode([df['Text'][i]])\n",
        "      candidate_embeddings = model.encode(candidates)\n",
        "      distances = cosine_similarity(doc_embedding, candidate_embeddings)\n",
        "      keywords = [candidates[index] for index in distances.argsort()[0][-top_n:]]\n",
        "\n",
        "      n_gram_range1 = (1, 1)\n",
        "      count1 = CountVectorizer(ngram_range=n_gram_range1, stop_words='english').fit(keywords)\n",
        "      candidates1 = count1.get_feature_names()\n",
        "\n",
        "      doc_embedding1 = model.encode(keywords)\n",
        "      candidate_embeddings1 = model.encode(candidates1)\n",
        "      distances1 = cosine_similarity(doc_embedding1, candidate_embeddings1)\n",
        "      a = [candidates1[index] for index in distances1.argsort()[0][-5:] if pos_tag(candidates1[index])[0][1] == 'NN' or pos_tag(candidates1[index])[0][1] == 'NNS' or pos_tag(candidates1[index])[0][1] == 'JJ']\n",
        "      if len(a) < 3:\n",
        "        a = [candidates1[index] for index in distances1.argsort()[0][-5:]]\n",
        "        b = [index for index in distances1.argsort()[0][-5:]]\n",
        "      k_word.append(a)\n",
        "      emb.append(b)\n",
        "    except ValueError:\n",
        "      k_word.append('No')\n",
        "      emb.append('')\n",
        "df['keyword']= k_word\n",
        "# df['emb']= emb"
      ],
      "metadata": {
        "id": "QcML2h-CAres"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df[['Text','keyword']][:10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 455
        },
        "id": "t-3jPR2x3IPl",
        "outputId": "d8a9b5c4-853c-4a9d-8c41-b0b84de8540b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-e643880e-5319-4cba-9123-1c72a52d053f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Text</th>\n",
              "      <th>keyword</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>I have bought several of the Vitality canned d...</td>\n",
              "      <td>['vitality', 'finicky', 'smells', 'labrador', ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
              "      <td>['jumbo', 'unsalted', 'small', 'sized', 'vendo...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>This is a confection that has been around a fe...</td>\n",
              "      <td>['gelatin', 'pillowy', 'mouthful', 'coated', '...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>If you are looking for the secret ingredient i...</td>\n",
              "      <td>['medicinal', 'robitussin', 'cherry', 'ingredi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Great taffy at a great price.  There was a wid...</td>\n",
              "      <td>['quick', 'wide', 'delivery', 'deal', 'lover']</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>I got a wild hair for taffy and ordered this f...</td>\n",
              "      <td>['treat', 'beer', 'melon', 'peppermint', 'poun...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>This saltwater taffy had great flavors and was...</td>\n",
              "      <td>['version', 'stuck', 'themed', 'fralinger', 'c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>This taffy is so good.  It is very soft and ch...</td>\n",
              "      <td>['taffy', 'soft', 'satisfying', 'amazing', 'ch...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Right now I'm mostly just sprouting this so my...</td>\n",
              "      <td>['mostly', 'sprouting', 'rye', 'grass', 'wheat...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>This is a very healthy dog food. Good for thei...</td>\n",
              "      <td>['digestion', 'healthy', 'feeding', 'small', '...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e643880e-5319-4cba-9123-1c72a52d053f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-e643880e-5319-4cba-9123-1c72a52d053f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-e643880e-5319-4cba-9123-1c72a52d053f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                                                Text                                            keyword\n",
              "0  I have bought several of the Vitality canned d...  ['vitality', 'finicky', 'smells', 'labrador', ...\n",
              "1  Product arrived labeled as Jumbo Salted Peanut...  ['jumbo', 'unsalted', 'small', 'sized', 'vendo...\n",
              "2  This is a confection that has been around a fe...  ['gelatin', 'pillowy', 'mouthful', 'coated', '...\n",
              "3  If you are looking for the secret ingredient i...  ['medicinal', 'robitussin', 'cherry', 'ingredi...\n",
              "4  Great taffy at a great price.  There was a wid...     ['quick', 'wide', 'delivery', 'deal', 'lover']\n",
              "5  I got a wild hair for taffy and ordered this f...  ['treat', 'beer', 'melon', 'peppermint', 'poun...\n",
              "6  This saltwater taffy had great flavors and was...  ['version', 'stuck', 'themed', 'fralinger', 'c...\n",
              "7  This taffy is so good.  It is very soft and ch...  ['taffy', 'soft', 'satisfying', 'amazing', 'ch...\n",
              "8  Right now I'm mostly just sprouting this so my...  ['mostly', 'sprouting', 'rye', 'grass', 'wheat...\n",
              "9  This is a very healthy dog food. Good for thei...  ['digestion', 'healthy', 'feeding', 'small', '..."
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### * 키워드 추출 모델 결과:  \n",
        "  * 형용사 키워드 추출: smells, unsalted, chewy 등  \n",
        "  * 명사 키워드 추출: stew, gelatin, delivery, digestion, meat 등\n",
        "  * 해당 키워드와 함께 된 scoring으로 상품 평 분석 가능\n"
      ],
      "metadata": {
        "id": "LWJIodoM4RdU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "### * 문제 및 개선점:\n",
        "  * 리뷰 내용이 매우 적거나 불용어만으로 이뤄진 경우 키워드가 나오지 않을 수 있음. 적은 수로 존재하기 때문에, No 라는 키워드로 반환\n",
        "  * 불용어가 100% 처리되지 않아 mostly 등과 같은 단어가 여전히 존재하므로 더 정교한 불용어 처리 필요"
      ],
      "metadata": {
        "id": "MUMJgfhWd_DH"
      }
    }
  ]
}