Classical Chinese Tutorial -- Counter as an Efficient Way to Do N-Gram.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classical Chinese Tutorial -- Counter as an Efficient Way to Do N-Gram\n",
    "\n",
    "Just found `Counter` is super useful for 1-gram frequency. For example, just input you txt:\n",
    "\n",
    "reference: https://stackoverflow.com/questions/43473736/most-common-2-grams-using-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('，', 12), ('之', 8), ('也', 8), ('。', 7), ('謂', 5)]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# just insert your string into Counter, it automatically count the 1-gram\n",
    "input_string = \"天命之謂性，率性之謂道，修道之謂教。道也者，不可須臾離也，可離非道也。是故君子戒慎乎其所不睹，恐懼乎其所不聞。莫見乎隱，莫顯乎微。故君子慎其獨也。喜怒哀樂之未發，謂之中；發而皆中節，謂之和；中也者，天下之大本也；和也者，天下之達道也。致中和，天地位焉，萬物育焉。\"\n",
    "one_gram = Counter(input_string)\n",
    "one_gram.most_common(5) # print most common 5 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, you want to get rid of the punctuations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('之', 8), ('也', 8), ('謂', 5), ('道', 5), ('天', 4)]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_string = re.sub(r'[，。？：「」；]', '', input_string)\n",
    "one_gram = Counter(clean_string)\n",
    "one_gram.most_common(5) # print most common 5 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How About Counting 2-Grams?\n",
    "\n",
    "Ok, we see one-gram is quite easy, but how about two-grams?  \n",
    "We can recall our memory about the usage of `zip`. `zip` is quite common if we want to loop over two list parallel in a for loop. We can use it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('之', '謂'), 3),\n",
       " (('道', '也'), 3),\n",
       " (('也', '者'), 3),\n",
       " (('故', '君'), 2),\n",
       " (('君', '子'), 2)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# counting two grams using zip\n",
    "two_gram = Counter(zip(clean_string, clean_string[1:]))\n",
    "two_gram.most_common(5) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Huh, it's neat."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3-Grams\n",
    "\n",
    "Just repeat what we have done..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('故', '君', '子'), 2),\n",
       " (('乎', '其', '所'), 2),\n",
       " (('其', '所', '不'), 2),\n",
       " (('也', '者', '天'), 2),\n",
       " (('者', '天', '下'), 2),\n",
       " (('天', '下', '之'), 2),\n",
       " (('天', '命', '之'), 1),\n",
       " (('命', '之', '謂'), 1),\n",
       " (('之', '謂', '性'), 1),\n",
       " (('謂', '性', '率'), 1)]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# counting two grams using zip\n",
    "three_gram = Counter(zip(clean_string, clean_string[1:], clean_string[2:]))\n",
    "three_gram.most_common(10) "
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}