Day 10 MarkovDecisionProcess_practice.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.4"
    },
    "toc": {
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": false
    },
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    },
    "colab": {
      "name": "Copy of MDP_practice.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lagom-QB/M12/blob/master/Day%2010%20MarkovDecisionProcess_practice.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XOqGd1mgsYPM",
        "colab_type": "text"
      },
      "source": [
        "## week07: Value based\n",
        "### Markov decision process\n",
        "_Reference: based on Practical RL_ [week02](https://github.com/yandexdataschool/Practical_RL/tree/master/week02_value_based)\n",
        "\n",
        "This week's methods are all built to solve __M__arkov __D__ecision __P__rocesses. In the broadest sense, an MDP is defined by how it changes states and how rewards are computed.\n",
        "\n",
        "State transition is defined by $P(s' |s,a)$ - how likely are you to end at state $s'$ if you take action $a$ from state $s$. Now there's more than one way to define rewards, but we'll use $r(s,a,s')$ function for convenience.\n",
        "\n",
        "_This notebook is inspired by the awesome_ [CS294](https://github.com/berkeleydeeprlcourse/homework/blob/36a0b58261acde756abd55306fbe63df226bf62b/hw2/HW2.ipynb) _by Berkeley_"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kEUuw2RnsYPR",
        "colab_type": "text"
      },
      "source": [
        "For starters, let's define a simple MDP from this picture:\n",
        "<img src='https://upload.wikimedia.org/wikipedia/commons/thumb/a/ad/Markov_Decision_Process.svg/800px-Markov_Decision_Process.svg.png' width=300px>\n",
        "_img by MistWiz (Own work) [Public domain], via Wikimedia Commons_"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:44:32.642838Z",
          "start_time": "2018-04-02T13:44:32.545142Z"
        },
        "id": "yaY80f0_sYPX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# If you Colab, uncomment this please\n",
        "!wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/spring19/week02_value_based/mdp.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:44:32.642838Z",
          "start_time": "2018-04-02T13:44:32.545142Z"
        },
        "id": "4YSQS8OusYPy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "transition_probs = {\n",
        "    's0': {\n",
        "        'a0': {'s0': 0.5, 's2': 0.5},\n",
        "        'a1': {'s2': 1}\n",
        "    },\n",
        "    's1': {\n",
        "        'a0': {'s0': 0.7, 's1': 0.1, 's2': 0.2},\n",
        "        'a1': {'s1': 0.95, 's2': 0.05}\n",
        "    },\n",
        "    's2': {\n",
        "        'a0': {'s0': 0.4, 's1': 0.6},\n",
        "        'a1': {'s0': 0.3, 's1': 0.3, 's2': 0.4}\n",
        "    }\n",
        "}\n",
        "rewards = {\n",
        "    's1': {'a0': {'s0': +5}},\n",
        "    's2': {'a1': {'s0': -1}}\n",
        "}\n",
        "\n",
        "from mdp import MDP\n",
        "mdp = MDP(transition_probs, rewards, initial_state='s0')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HiTp2xcOsYP-",
        "colab_type": "text"
      },
      "source": [
        "We can now use MDP just as any other gym environment:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:44:34.203384Z",
          "start_time": "2018-04-02T13:44:34.199297Z"
        },
        "id": "rdIvSOjysYQC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 40
        },
        "outputId": "9505c04c-7178-43ae-b91b-f2d7c6d54737"
      },
      "source": [
        "print('initial state =', mdp.reset())\n",
        "next_state, reward, done, info = mdp.step('a1')\n",
        "print('next_state = %s, reward = %s, done = %s' % (next_state, reward, done))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "initial state = s0\n",
            "next_state = s2, reward = 0.0, done = False\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kq2AGvSZsYQP",
        "colab_type": "text"
      },
      "source": [
        "but it also has other methods that you'll need for Value Iteration"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:44:34.956122Z",
          "start_time": "2018-04-02T13:44:34.949856Z"
        },
        "id": "gCBhHQD0sYQV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 75
        },
        "outputId": "c1b16b9d-2c4d-42a2-cbbc-66f9c788f78a"
      },
      "source": [
        "print(\"mdp.get_all_states =\", mdp.get_all_states())\n",
        "print(\"mdp.get_possible_actions('s1') = \", mdp.get_possible_actions('s1'))\n",
        "print(\"mdp.get_next_states('s1', 'a0') = \", mdp.get_next_states('s1', 'a0'))\n",
        "print(\"mdp.get_reward('s1', 'a0', 's0') = \", mdp.get_reward('s1', 'a0', 's0'))\n",
        "print(\"mdp.get_transition_prob('s1', 'a0', 's0') = \",\n",
        "      mdp.get_transition_prob('s1', 'a0', 's0'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mdp.get_all_states = ('s0', 's1', 's2')\n",
            "mdp.get_possible_actions('s1') =  ('a0', 'a1')\n",
            "mdp.get_next_states('s1', 'a0') =  {'s0': 0.7, 's1': 0.1, 's2': 0.2}\n",
            "mdp.get_reward('s1', 'a0', 's0') =  5\n",
            "mdp.get_transition_prob('s1', 'a0', 's0') =  0.7\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qxSxh7DasYQf",
        "colab_type": "text"
      },
      "source": [
        "### Optional: Visualizing MDPs\n",
        "\n",
        "You can also visualize any MDP with the drawing fuction donated by [neer201](https://github.com/neer201).\n",
        "\n",
        "You have to install graphviz for system and for python. For __ubuntu__ just run:\n",
        "\n",
        "1. `sudo apt-get install graphviz`\n",
        "2. `pip install graphviz`\n",
        "3. restart the notebook\n",
        "\n",
        "For __macOs__ use `brew` instead of `sudo apt-get`. \n",
        "\n",
        "__Note:__ Installing graphviz on some OS (esp. Windows) may be tricky. However, you can ignore this part alltogether and use the standart vizualization."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:44:37.797182Z",
          "start_time": "2018-04-02T13:44:37.794073Z"
        },
        "id": "jZtKMEJtsYQh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 29
        },
        "outputId": "8f2dc85f-369e-4e0d-fe54-bf0a308cefc2"
      },
      "source": [
        "from mdp import has_graphviz\n",
        "from IPython.display import display\n",
        "print(\"Graphviz available:\", has_graphviz)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Graphviz available: True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:44:38.715883Z",
          "start_time": "2018-04-02T13:44:38.648684Z"
        },
        "id": "5pFLSQV4sYQu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "29f1d9da-13eb-44bf-d8f0-718ea93e18fb"
      },
      "source": [
        "if has_graphviz:\n",
        "    from mdp import plot_graph, plot_graph_with_state_values, \\\n",
        "        plot_graph_optimal_strategy_and_state_values\n",
        "\n",
        "    display(plot_graph(mdp))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7f0ef0bdfcf8>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n -->\n<!-- Title: MDP Pages: 1 -->\n<svg width=\"720pt\" height=\"196pt\"\n viewBox=\"0.00 0.00 720.00 195.51\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(.7008 .7008) rotate(0) translate(4 275)\">\n<title>MDP</title>\n<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-275 1023.463,-275 1023.463,4 -4,4\"/>\n<!-- s0 -->\n<g id=\"node1\" class=\"node\">\n<title>s0</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"40\" cy=\"-102\" rx=\"36\" ry=\"36\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"40\" cy=\"-102\" rx=\"40\" ry=\"40\"/>\n<text text-anchor=\"middle\" x=\"40\" y=\"-95.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s0</text>\n</g>\n<!-- s0&#45;a0 -->\n<g id=\"node2\" class=\"node\">\n<title>s0&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"193.5772\" cy=\"-146\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"193.5772\" y=\"-141\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s0&#45;&gt;s0&#45;a0 -->\n<g id=\"edge1\" class=\"edge\">\n<title>s0&#45;&gt;s0&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M79.2844,-110.7026C99.8974,-115.5498 125.4811,-122.0095 148,-129 151.327,-130.0328 154.7652,-131.1741 158.1924,-132.3622\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"157.1257,-135.698 167.7202,-135.7831 159.4912,-129.1097 157.1257,-135.698\"/>\n</g>\n<!-- s0&#45;a1 -->\n<g id=\"node4\" class=\"node\">\n<title>s0&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"193.5772\" cy=\"-219\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"193.5772\" y=\"-214\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s0&#45;&gt;s0&#45;a1 -->\n<g id=\"edge4\" class=\"edge\">\n<title>s0&#45;&gt;s0&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M66.6878,-131.9984C76.104,-141.7862 87.0765,-152.3514 98,-161 117.3546,-176.324 141.0693,-190.7377 160.0344,-201.3632\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"158.4757,-204.5005 168.923,-206.2624 161.8547,-198.37 158.4757,-204.5005\"/>\n</g>\n<!-- s0&#45;a0&#45;&gt;s0 -->\n<g id=\"edge2\" class=\"edge\">\n<title>s0&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M166.0433,-141.6398C146.7591,-138.2473 120.5165,-132.9481 98,-126 94.1846,-124.8227 90.2733,-123.4907 86.3715,-122.0724\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"87.3439,-118.6975 76.753,-118.4084 84.852,-125.239 87.3439,-118.6975\"/>\n<text text-anchor=\"middle\" x=\"123\" y=\"-144.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.5</text>\n</g>\n<!-- s2 -->\n<g id=\"node3\" class=\"node\">\n<title>s2</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"433.1543\" cy=\"-158\" rx=\"36\" ry=\"36\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"433.1543\" cy=\"-158\" rx=\"40\" ry=\"40\"/>\n<text text-anchor=\"middle\" x=\"433.1543\" y=\"-151.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s2</text>\n</g>\n<!-- s0&#45;a0&#45;&gt;s2 -->\n<g id=\"edge3\" class=\"edge\">\n<title>s0&#45;a0&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M221.5286,-147.4C261.0547,-149.3798 333.9202,-153.0295 382.9604,-155.4859\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"382.8803,-158.9862 393.0429,-155.9909 383.2305,-151.9949 382.8803,-158.9862\"/>\n<text text-anchor=\"middle\" x=\"307.1543\" y=\"-160.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.5</text>\n</g>\n<!-- s2&#45;a0 -->\n<g id=\"node8\" class=\"node\">\n<title>s2&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"666.7315\" cy=\"-129\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"666.7315\" y=\"-124\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s2&#45;&gt;s2&#45;a0 -->\n<g id=\"edge13\" class=\"edge\">\n<title>s2&#45;&gt;s2&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M473.5103,-157.8232C511.8415,-156.9917 570.9151,-153.9822 621.1543,-144 624.3408,-143.3669 627.6142,-142.5785 630.8741,-141.6969\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"631.9229,-145.0366 640.5126,-138.8344 629.93,-138.3263 631.9229,-145.0366\"/>\n</g>\n<!-- s2&#45;a1 -->\n<g id=\"node9\" class=\"node\">\n<title>s2&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"666.7315\" cy=\"-56\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"666.7315\" y=\"-51\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s2&#45;&gt;s2&#45;a1 -->\n<g id=\"edge16\" class=\"edge\">\n<title>s2&#45;&gt;s2&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M459.0758,-127.434C468.3559,-118.1046 479.4406,-108.5837 491.1543,-102 534.782,-77.4792 591.8554,-65.6786 628.8989,-60.2529\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"629.8226,-63.6588 639.2512,-58.8264 628.867,-56.7243 629.8226,-63.6588\"/>\n</g>\n<!-- s0&#45;a1&#45;&gt;s2 -->\n<g id=\"edge5\" class=\"edge\">\n<title>s0&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M220.8397,-213.4876C256.5286,-206.0905 320.9236,-192.1221 375.1543,-177 378.5225,-176.0608 381.9861,-175.048 385.4645,-173.9962\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"386.7297,-177.2682 395.2399,-170.9575 384.6517,-170.5837 386.7297,-177.2682\"/>\n<text text-anchor=\"middle\" x=\"307.1543\" y=\"-213.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 1</text>\n</g>\n<!-- s1 -->\n<g id=\"node5\" class=\"node\">\n<title>s1</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"829.3087\" cy=\"-106\" rx=\"36\" ry=\"36\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"829.3087\" cy=\"-106\" rx=\"40\" ry=\"40\"/>\n<text text-anchor=\"middle\" x=\"829.3087\" y=\"-99.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s1</text>\n</g>\n<!-- s1&#45;a0 -->\n<g id=\"node6\" class=\"node\">\n<title>s1&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"991.8858\" cy=\"-70\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"991.8858\" y=\"-65\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s1&#45;&gt;s1&#45;a0 -->\n<g id=\"edge6\" class=\"edge\">\n<title>s1&#45;&gt;s1&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M868.6641,-97.2854C894.7641,-91.506 928.8065,-83.9679 954.4945,-78.2797\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"955.5871,-81.6226 964.5939,-76.0433 954.0737,-74.7882 955.5871,-81.6226\"/>\n</g>\n<!-- s1&#45;a1 -->\n<g id=\"node7\" class=\"node\">\n<title>s1&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"991.8858\" cy=\"-152\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"991.8858\" y=\"-147\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s1&#45;&gt;s1&#45;a1 -->\n<g id=\"edge10\" class=\"edge\">\n<title>s1&#45;&gt;s1&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M868.7555,-113.3782C891.6544,-118.0876 920.9044,-124.8376 946.3087,-133 949.8641,-134.1424 953.5293,-135.4461 957.1639,-136.8233\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"956.0315,-140.1394 966.6171,-140.5796 958.6164,-133.6342 956.0315,-140.1394\"/>\n</g>\n<!-- s1&#45;a0&#45;&gt;s0 -->\n<g id=\"edge7\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M969.5089,-53.411C939.3125,-32.711 883.0867,0 829.3087,0 193.5772,0 193.5772,0 193.5772,0 145.8315,0 101.5836,-34.6324 72.9684,-63.537\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"70.0864,-61.4847 65.6803,-71.1198 75.1333,-66.3354 70.0864,-61.4847\"/>\n<text text-anchor=\"middle\" x=\"556.1543\" y=\"-5.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.7 &#160;reward =5</text>\n</g>\n<!-- s1&#45;a0&#45;&gt;s2 -->\n<g id=\"edge9\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M976.3243,-93.2984C972.1185,-100.1282 967.7762,-107.7204 964.3087,-115 953.9716,-136.7008 965.931,-152.1169 946.3087,-166 873.0415,-217.8376 597.2942,-183.133 483.1269,-165.993\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"483.3302,-162.4838 472.918,-164.4422 482.2789,-169.4044 483.3302,-162.4838\"/>\n<text text-anchor=\"middle\" x=\"741.8087\" y=\"-198.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.2</text>\n</g>\n<!-- s1&#45;a0&#45;&gt;s1 -->\n<g id=\"edge8\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M965.7872,-61.0803C944.1683,-55.1324 912.9482,-49.9157 887.3087,-59 879.6639,-61.7086 872.2908,-65.9014 865.5135,-70.691\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"863.0781,-68.1479 857.2719,-77.0101 867.3373,-73.703 863.0781,-68.1479\"/>\n<text text-anchor=\"middle\" x=\"916.8087\" y=\"-64.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.1</text>\n</g>\n<!-- s1&#45;a1&#45;&gt;s2 -->\n<g id=\"edge12\" class=\"edge\">\n<title>s1&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M975.9063,-175.015C968.1402,-184.234 957.96,-193.8712 946.3087,-199 761.1617,-280.4996 683.2412,-262.4365 491.1543,-199 484.6246,-196.8436 478.1874,-193.6811 472.1124,-190.0532\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"473.7965,-186.9738 463.5022,-184.4679 469.987,-192.8464 473.7965,-186.9738\"/>\n<text text-anchor=\"middle\" x=\"741.8087\" y=\"-258.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.05</text>\n</g>\n<!-- s1&#45;a1&#45;&gt;s1 -->\n<g id=\"edge11\" class=\"edge\">\n<title>s1&#45;a1&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M964.4489,-147.6602C943.147,-143.9487 913.0104,-137.9461 887.3087,-130 883.4939,-128.8206 879.583,-127.4872 875.6816,-126.068\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"876.6544,-122.6932 866.0635,-122.4023 874.1614,-129.2342 876.6544,-122.6932\"/>\n<text text-anchor=\"middle\" x=\"916.8087\" y=\"-149.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.95</text>\n</g>\n<!-- s2&#45;a0&#45;&gt;s0 -->\n<g id=\"edge14\" class=\"edge\">\n<title>s2&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M643.5438,-113.3955C636.633,-109.559 628.8443,-105.9847 621.1543,-104 429.892,-54.638 191.2809,-79.6056 89.5999,-94.0846\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"89.0082,-90.6337 79.6156,-95.5358 90.0151,-97.561 89.0082,-90.6337\"/>\n<text text-anchor=\"middle\" x=\"307.1543\" y=\"-84.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.4</text>\n</g>\n<!-- s2&#45;a0&#45;&gt;s1 -->\n<g id=\"edge15\" class=\"edge\">\n<title>s2&#45;a0&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M694.3105,-125.0984C717.6285,-121.7995 751.619,-116.9909 779.6586,-113.0241\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"780.2778,-116.4714 789.6889,-111.6051 779.2972,-109.5404 780.2778,-116.4714\"/>\n<text text-anchor=\"middle\" x=\"741.8087\" y=\"-128.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.6</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s0 -->\n<g id=\"edge17\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M639.1151,-53.6811C570.962,-48.4197 389.3931,-37.211 239.1543,-53 175.3691,-59.7034 159.9312,-66.3263 98,-83 94.6236,-83.909 91.1538,-84.8994 87.671,-85.9351\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"86.4168,-82.659 77.8873,-88.9438 88.4744,-89.3497 86.4168,-82.659\"/>\n<text text-anchor=\"middle\" x=\"307.1543\" y=\"-58.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.3 &#160;reward =&#45;1</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s2 -->\n<g id=\"edge19\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M648.0919,-76.9874C640.4469,-84.3829 631.0663,-92.0732 621.1543,-97 568.4674,-123.188 545.9506,-100.5736 491.1543,-122 485.639,-124.1566 480.0669,-126.8429 474.6705,-129.7699\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"472.7205,-126.8538 465.7954,-134.8722 476.2093,-132.9224 472.7205,-126.8538\"/>\n<text text-anchor=\"middle\" x=\"556.1543\" y=\"-127.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.4</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s1 -->\n<g id=\"edge18\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M693.2142,-64.1446C716.9579,-71.4469 752.3787,-82.3405 781.1324,-91.1836\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"780.4078,-94.6224 790.9948,-94.2167 782.4655,-87.9317 780.4078,-94.6224\"/>\n<text text-anchor=\"middle\" x=\"741.8087\" y=\"-92.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.3</text>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NBsJVsSHsYQ4",
        "colab_type": "text"
      },
      "source": [
        "### Value Iteration\n",
        "\n",
        "Now let's build something to solve this MDP. The simplest algorithm so far is __V__alue __I__teration\n",
        "\n",
        "Here's the pseudo-code for VI:\n",
        "\n",
        "---\n",
        "\n",
        "`1.` Initialize $V^{(0)}(s)=0$, for all $s$\n",
        "\n",
        "`2.` For $i=0, 1, 2, \\dots$\n",
        " \n",
        "`3.` $ \\quad V_{(i+1)}(s) = \\max_a \\sum_{s'} P(s' | s,a) \\cdot [ r(s,a,s') + \\gamma V_{i}(s')]$, for all $s$\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ghf3aI-sYQ5",
        "colab_type": "text"
      },
      "source": [
        "First, let's write a function to compute the state-action value function $Q^{\\pi}$, defined as follows\n",
        "\n",
        "$$Q_i(s, a) = \\sum_{s'} P(s' | s,a) \\cdot [ r(s,a,s') + \\gamma V_{i}(s')]$$\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.101416Z",
          "start_time": "2018-04-02T13:43:17.095468Z"
        },
        "id": "qMTRPgFZsYQ6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 29
        },
        "outputId": "3c777027-46b2-4f93-df4e-77162d7251ce"
      },
      "source": [
        "%%writefile mdp_get_action_value.py\n",
        "\n",
        "def get_action_value(mdp, state_values, state, action, gamma):\n",
        "    \"\"\" Computes Q(s,a) as in formula above \"\"\"\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "    results = 0.\n",
        "    for posible_state in mdp.get_all_states():\n",
        "        transition_prob = mdp.get_transition_prob(state, action, posible_state)\n",
        "        reward = mdp.get_reward(state, action, posible_state)\n",
        "        state_value = state_values[posible_state]\n",
        " \n",
        "        results += transition_prob * (reward + gamma * state_value)\n",
        "\n",
        "    return results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Overwriting mdp_get_action_value.py\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RuY-v894sYRB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from mdp_get_action_value import get_action_value"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.102247Z",
          "start_time": "2018-04-02T13:43:05.502Z"
        },
        "id": "EthZMP3dsYRL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "test_Vs = {s: i for i, s in enumerate(sorted(mdp.get_all_states()))}\n",
        "assert np.allclose(get_action_value(mdp, test_Vs, 's2', 'a1', 0.9), 0.69)\n",
        "assert np.allclose(get_action_value(mdp, test_Vs, 's1', 'a0', 0.9), 3.95)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yTjVrJ5CsYRZ",
        "colab_type": "text"
      },
      "source": [
        "Using $Q(s,a)$ we can now define the \"next\" V(s) for value iteration.\n",
        " $$V_{(i+1)}(s) = \\max_a \\sum_{s'} P(s' | s,a) \\cdot [ r(s,a,s') + \\gamma V_{i}(s')] = \\max_a Q_i(s,a)$$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.103358Z",
          "start_time": "2018-04-02T13:43:05.506Z"
        },
        "id": "XNUrgq-asYRb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_new_state_value(mdp, state_values, state, gamma):\n",
        "    \"\"\" Computes next V(s) as in formula above. Please do not change state_values in process. \"\"\"\n",
        "    if mdp.is_terminal(state):\n",
        "        return 0\n",
        "   \n",
        "    max_value = -np.inf\n",
        "    result = None\n",
        "   \n",
        "    for possible_action in mdp.get_possible_actions(state):\n",
        "        q_sa = get_action_value(mdp, state_values, state, possible_action, gamma)\n",
        "        if q_sa > max_value:\n",
        "            max_value = q_sa\n",
        "    # <YOUR CODE>\n",
        "    return max_value"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.104340Z",
          "start_time": "2018-04-02T13:43:05.510Z"
        },
        "id": "fbRVeHUasYRj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "test_Vs_copy = dict(test_Vs)\n",
        "assert np.allclose(get_new_state_value(mdp, test_Vs, 's0', 0.9), 1.8)\n",
        "assert np.allclose(get_new_state_value(mdp, test_Vs, 's2', 0.9), 0.69)\n",
        "assert test_Vs == test_Vs_copy, \"please do not change state_values in get_new_state_value\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3um1AiMHsYRu",
        "colab_type": "text"
      },
      "source": [
        "Finally, let's combine everything we wrote into a working value iteration algo."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:44:09.793405Z",
          "start_time": "2018-04-02T13:44:09.770623Z"
        },
        "id": "3ynhjguCsYRw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "2d666537-0ad1-4707-daf2-e4456c8c6e45"
      },
      "source": [
        "# parameters\n",
        "gamma = 0.9            # discount for MDP\n",
        "num_iter = 100         # maximum iterations, excluding initialization\n",
        "# stop VI if new values are this close to old values (or closer)\n",
        "min_difference = 0.001\n",
        "\n",
        "# initialize V(s)\n",
        "state_values = {s: 0 for s in mdp.get_all_states()}\n",
        "\n",
        "if has_graphviz:\n",
        "    display(plot_graph_with_state_values(mdp, state_values))\n",
        "\n",
        "for i in range(num_iter):\n",
        "\n",
        "    # Compute new state values using the functions you defined above.\n",
        "    # It must be a dict {state : float V_new(state)}\n",
        "    new_state_values = {\n",
        "            _s: get_new_state_value(mdp, state_values, _s, gamma) # <YOUR CODE HERE >\n",
        "            for _s in state_values.keys()\n",
        "        }\n",
        "    assert isinstance(new_state_values, dict)\n",
        "\n",
        "    # Compute difference\n",
        "    diff = max(abs(new_state_values[s] - state_values[s])\n",
        "               for s in mdp.get_all_states())\n",
        "    print(\"iter %4i   |   diff: %6.5f   |   \" % (i, diff), end=\"\")\n",
        "    print('   '.join(\"V(%s) = %.3f\" % (s, v)\n",
        "                     for s, v in state_values.items()), end='\\n\\n')\n",
        "    state_values = new_state_values\n",
        "\n",
        "    if diff < min_difference:\n",
        "        print(\"Terminated\")\n",
        "        break"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7f0eefdf0470>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n -->\n<!-- Title: MDP Pages: 1 -->\n<svg width=\"720pt\" height=\"180pt\"\n viewBox=\"0.00 0.00 720.00 180.05\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(.6571 .6571) rotate(0) translate(4 270)\">\n<title>MDP</title>\n<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-270 1091.7199,-270 1091.7199,4 -4,4\"/>\n<!-- s0 -->\n<g id=\"node1\" class=\"node\">\n<title>s0</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"51.3762\" cy=\"-103\" rx=\"47.2622\" ry=\"47.2622\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"51.3762\" cy=\"-103\" rx=\"51.2526\" ry=\"51.2526\"/>\n<text text-anchor=\"middle\" x=\"51.3762\" y=\"-109.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s0</text>\n<text text-anchor=\"middle\" x=\"51.3762\" y=\"-83.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">V =0</text>\n</g>\n<!-- s0&#45;a0 -->\n<g id=\"node2\" class=\"node\">\n<title>s0&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"216.3295\" cy=\"-147\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"216.3295\" y=\"-142\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s0&#45;&gt;s0&#45;a0 -->\n<g id=\"edge1\" class=\"edge\">\n<title>s0&#45;&gt;s0&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M102.0079,-112.8759C123.3915,-117.4524 148.4549,-123.3529 170.7523,-130 174.0908,-130.9952 177.5367,-132.1111 180.9688,-133.2829\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"179.9093,-136.6209 190.504,-136.6798 182.2584,-130.0268 179.9093,-136.6209\"/>\n</g>\n<!-- s0&#45;a1 -->\n<g id=\"node4\" class=\"node\">\n<title>s0&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"216.3295\" cy=\"-220\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"216.3295\" y=\"-215\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s0&#45;&gt;s0&#45;a1 -->\n<g id=\"edge4\" class=\"edge\">\n<title>s0&#45;&gt;s0&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M90.2161,-137.1326C99.9695,-145.2363 110.5648,-153.6548 120.7523,-161 140.8702,-175.5051 164.5898,-190.1605 183.3665,-201.2365\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"181.7499,-204.3456 192.1496,-206.3699 185.2822,-198.3021 181.7499,-204.3456\"/>\n</g>\n<!-- s0&#45;a0&#45;&gt;s0 -->\n<g id=\"edge2\" class=\"edge\">\n<title>s0&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M188.9184,-142.2197C169.6906,-138.6345 143.4649,-133.2777 120.7523,-127 117.163,-126.0079 113.4893,-124.9297 109.7972,-123.7978\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"110.6165,-120.3859 100.0263,-120.699 108.5003,-127.0584 110.6165,-120.3859\"/>\n<text text-anchor=\"middle\" x=\"145.7523\" y=\"-144.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.5</text>\n</g>\n<!-- s2 -->\n<g id=\"node3\" class=\"node\">\n<title>s2</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"467.2828\" cy=\"-170\" rx=\"47.2622\" ry=\"47.2622\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"467.2828\" cy=\"-170\" rx=\"51.2526\" ry=\"51.2526\"/>\n<text text-anchor=\"middle\" x=\"467.2828\" y=\"-176.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s2</text>\n<text text-anchor=\"middle\" x=\"467.2828\" y=\"-150.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">V =0</text>\n</g>\n<!-- s0&#45;a0&#45;&gt;s2 -->\n<g id=\"edge3\" class=\"edge\">\n<title>s0&#45;a0&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M244.1447,-149.5493C282.9077,-153.1019 354.2131,-159.6371 405.8055,-164.3656\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"405.5679,-167.8584 415.8457,-165.2858 406.2069,-160.8876 405.5679,-167.8584\"/>\n<text text-anchor=\"middle\" x=\"329.9066\" y=\"-169.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.5</text>\n</g>\n<!-- s2&#45;a0 -->\n<g id=\"node8\" class=\"node\">\n<title>s2&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"712.2361\" cy=\"-129\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"712.2361\" y=\"-124\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s2&#45;&gt;s2&#45;a0 -->\n<g id=\"edge13\" class=\"edge\">\n<title>s2&#45;&gt;s2&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M518.621,-166.5831C559.2052,-163.16 617.0818,-156.6697 666.6589,-145 670.0262,-144.2074 673.4916,-143.2548 676.9362,-142.2144\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"678.0613,-145.529 686.4898,-139.1094 675.8976,-138.8718 678.0613,-145.529\"/>\n</g>\n<!-- s2&#45;a1 -->\n<g id=\"node9\" class=\"node\">\n<title>s2&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"712.2361\" cy=\"-56\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"712.2361\" y=\"-51\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s2&#45;&gt;s2&#45;a1 -->\n<g id=\"edge16\" class=\"edge\">\n<title>s2&#45;&gt;s2&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M500.7741,-131.03C511.3505,-120.6703 523.6878,-110.3369 536.6589,-103 580.2812,-78.3258 637.3558,-66.1773 674.4012,-60.5095\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"675.3568,-63.9079 684.754,-59.0147 674.3564,-56.9798 675.3568,-63.9079\"/>\n</g>\n<!-- s0&#45;a1&#45;&gt;s2 -->\n<g id=\"edge5\" class=\"edge\">\n<title>s0&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M243.7775,-215.2269C279.3968,-208.9489 343.4374,-197.3755 397.9066,-186 400.8956,-185.3758 403.9537,-184.723 407.0398,-184.0531\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"407.9297,-187.441 416.9409,-181.8688 406.4216,-180.6054 407.9297,-187.441\"/>\n<text text-anchor=\"middle\" x=\"329.9066\" y=\"-216.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 1</text>\n</g>\n<!-- s1 -->\n<g id=\"node5\" class=\"node\">\n<title>s1</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"886.1894\" cy=\"-100\" rx=\"47.2622\" ry=\"47.2622\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"886.1894\" cy=\"-100\" rx=\"51.2526\" ry=\"51.2526\"/>\n<text text-anchor=\"middle\" x=\"886.1894\" y=\"-106.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s1</text>\n<text text-anchor=\"middle\" x=\"886.1894\" y=\"-80.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">V =0</text>\n</g>\n<!-- s1&#45;a0 -->\n<g id=\"node6\" class=\"node\">\n<title>s1&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"1060.1427\" cy=\"-70\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"1060.1427\" y=\"-65\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s1&#45;&gt;s1&#45;a0 -->\n<g id=\"edge6\" class=\"edge\">\n<title>s1&#45;&gt;s1&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M937.228,-92.7844C960.822,-89.2603 989.1805,-84.7576 1014.5656,-80 1017.4021,-79.4684 1020.3319,-78.8911 1023.2739,-78.2911\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"1024.1435,-81.6848 1033.2064,-76.1972 1022.6995,-74.8353 1024.1435,-81.6848\"/>\n</g>\n<!-- s1&#45;a1 -->\n<g id=\"node7\" class=\"node\">\n<title>s1&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"1060.1427\" cy=\"-152\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"1060.1427\" y=\"-147\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s1&#45;&gt;s1&#45;a1 -->\n<g id=\"edge10\" class=\"edge\">\n<title>s1&#45;&gt;s1&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M936.3597,-111.1306C960.2753,-116.8837 989.1453,-124.4831 1014.5656,-133 1018.1066,-134.1864 1021.7623,-135.5192 1025.391,-136.9142\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"1024.2518,-140.2282 1034.8364,-140.6945 1026.8528,-133.7293 1024.2518,-140.2282\"/>\n</g>\n<!-- s1&#45;a0&#45;&gt;s0 -->\n<g id=\"edge7\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1037.4039,-54.2906C1005.2541,-33.6523 943.9723,0 886.1894,0 216.3295,0 216.3295,0 216.3295,0 169.4871,0 124.9678,-30.2534 94.0122,-57.9679\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"91.4267,-55.5907 86.4412,-64.9392 96.1684,-60.7402 91.4267,-55.5907\"/>\n<text text-anchor=\"middle\" x=\"601.6589\" y=\"-5.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.7 &#160;reward =5</text>\n</g>\n<!-- s1&#45;a0&#45;&gt;s2 -->\n<g id=\"edge9\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1044.5813,-93.2984C1040.3754,-100.1282 1036.0332,-107.7204 1032.5656,-115 1022.2285,-136.7008 1034.1279,-152.0324 1014.5656,-166 938.3171,-220.4417 654.7752,-193.3803 528.5583,-178.0786\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"528.6467,-174.5633 518.2946,-176.8181 527.7934,-181.5111 528.6467,-174.5633\"/>\n<text text-anchor=\"middle\" x=\"787.3133\" y=\"-203.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.2</text>\n</g>\n<!-- s1&#45;a0&#45;&gt;s1 -->\n<g id=\"edge8\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1033.4837,-61.3982C1012.0818,-55.8066 981.4407,-50.7779 955.5656,-58 949.4248,-59.714 943.2859,-62.1357 937.3403,-64.961\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"935.463,-61.9913 928.1837,-69.6896 938.6749,-68.2109 935.463,-61.9913\"/>\n<text text-anchor=\"middle\" x=\"985.0656\" y=\"-63.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.1</text>\n</g>\n<!-- s1&#45;a1&#45;&gt;s2 -->\n<g id=\"edge12\" class=\"edge\">\n<title>s1&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1044.1712,-175.0331C1036.4066,-184.2556 1026.2253,-193.8904 1014.5656,-199 820.0232,-284.2542 744.4278,-243.1259 536.6589,-199 532.3033,-198.075 527.8767,-196.8816 523.4755,-195.5078\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"524.5126,-192.1636 513.918,-192.2463 522.2518,-198.7885 524.5126,-192.1636\"/>\n<text text-anchor=\"middle\" x=\"787.3133\" y=\"-253.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.05</text>\n</g>\n<!-- s1&#45;a1&#45;&gt;s1 -->\n<g id=\"edge11\" class=\"edge\">\n<title>s1&#45;a1&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1032.67,-147.7746C1011.3487,-144.1246 981.2038,-138.1489 955.5656,-130 951.2999,-128.6442 946.9393,-127.1171 942.5849,-125.4884\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"943.6893,-122.1624 933.1011,-121.7878 941.1447,-128.6835 943.6893,-122.1624\"/>\n<text text-anchor=\"middle\" x=\"985.0656\" y=\"-149.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.95</text>\n</g>\n<!-- s2&#45;a0&#45;&gt;s0 -->\n<g id=\"edge14\" class=\"edge\">\n<title>s2&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M688.9795,-114.0141C682.0675,-110.3309 674.2955,-106.9009 666.6589,-105 468.8875,-55.7696 224.148,-78.8099 112.3029,-93.7412\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"111.6586,-90.2966 102.222,-95.1135 112.6028,-97.2326 111.6586,-90.2966\"/>\n<text text-anchor=\"middle\" x=\"329.9066\" y=\"-84.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.4</text>\n</g>\n<!-- s2&#45;a0&#45;&gt;s1 -->\n<g id=\"edge15\" class=\"edge\">\n<title>s2&#45;a0&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M739.7995,-124.4049C762.6806,-120.5903 796.0699,-115.0239 825.1845,-110.1702\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"825.9919,-113.584 835.2802,-108.4872 824.8407,-106.6793 825.9919,-113.584\"/>\n<text text-anchor=\"middle\" x=\"787.3133\" y=\"-127.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.6</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s0 -->\n<g id=\"edge17\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M684.7176,-53.799C614.1905,-48.6081 421.3039,-37.2485 261.9066,-54 209.9061,-59.4649 152.1463,-73.4958 110.2997,-85.1497\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"109.2264,-81.8158 100.5534,-87.901 111.1282,-88.5525 109.2264,-81.8158\"/>\n<text text-anchor=\"middle\" x=\"329.9066\" y=\"-59.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.3 &#160;reward =&#45;1</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s2 -->\n<g id=\"edge19\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M693.5771,-76.9486C685.9297,-84.3393 676.5526,-92.0366 666.6589,-97 613.9929,-123.4213 591.0115,-100.2492 536.6589,-123 529.9238,-125.8192 523.163,-129.3394 516.6456,-133.1774\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"514.5934,-130.3314 507.9342,-138.572 518.2788,-136.2827 514.5934,-130.3314\"/>\n<text text-anchor=\"middle\" x=\"601.6589\" y=\"-128.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.4</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s1 -->\n<g id=\"edge18\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M739.0351,-62.7786C762.2392,-68.6479 796.6516,-77.3522 826.365,-84.8679\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"825.6856,-88.3062 836.2385,-87.3654 827.4021,-81.52 825.6856,-88.3062\"/>\n<text text-anchor=\"middle\" x=\"787.3133\" y=\"-87.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.3</text>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "iter    0   |   diff: 3.50000   |   V(s0) = 0.000   V(s1) = 0.000   V(s2) = 0.000\n",
            "\n",
            "iter    1   |   diff: 1.89000   |   V(s0) = 0.000   V(s1) = 3.500   V(s2) = 0.000\n",
            "\n",
            "iter    2   |   diff: 1.70100   |   V(s0) = 0.000   V(s1) = 3.815   V(s2) = 1.890\n",
            "\n",
            "iter    3   |   diff: 1.13542   |   V(s0) = 1.701   V(s1) = 4.184   V(s2) = 2.060\n",
            "\n",
            "iter    4   |   diff: 0.73024   |   V(s0) = 1.854   V(s1) = 5.319   V(s2) = 2.871\n",
            "\n",
            "iter    5   |   diff: 0.61135   |   V(s0) = 2.584   V(s1) = 5.664   V(s2) = 3.540\n",
            "\n",
            "iter    6   |   diff: 0.54664   |   V(s0) = 3.186   V(s1) = 6.275   V(s2) = 3.989\n",
            "\n",
            "iter    7   |   diff: 0.49198   |   V(s0) = 3.590   V(s1) = 6.790   V(s2) = 4.535\n",
            "\n",
            "iter    8   |   diff: 0.42210   |   V(s0) = 4.082   V(s1) = 7.189   V(s2) = 4.959\n",
            "\n",
            "iter    9   |   diff: 0.36513   |   V(s0) = 4.463   V(s1) = 7.611   V(s2) = 5.352\n",
            "\n",
            "iter   10   |   diff: 0.32862   |   V(s0) = 4.816   V(s1) = 7.960   V(s2) = 5.717\n",
            "\n",
            "iter   11   |   diff: 0.29262   |   V(s0) = 5.145   V(s1) = 8.280   V(s2) = 6.032\n",
            "\n",
            "iter   12   |   diff: 0.26189   |   V(s0) = 5.429   V(s1) = 8.572   V(s2) = 6.323\n",
            "\n",
            "iter   13   |   diff: 0.23503   |   V(s0) = 5.691   V(s1) = 8.830   V(s2) = 6.584\n",
            "\n",
            "iter   14   |   diff: 0.21124   |   V(s0) = 5.925   V(s1) = 9.065   V(s2) = 6.817\n",
            "\n",
            "iter   15   |   diff: 0.19012   |   V(s0) = 6.135   V(s1) = 9.276   V(s2) = 7.028\n",
            "\n",
            "iter   16   |   diff: 0.17091   |   V(s0) = 6.325   V(s1) = 9.465   V(s2) = 7.218\n",
            "\n",
            "iter   17   |   diff: 0.15366   |   V(s0) = 6.496   V(s1) = 9.636   V(s2) = 7.388\n",
            "\n",
            "iter   18   |   diff: 0.13830   |   V(s0) = 6.649   V(s1) = 9.790   V(s2) = 7.542\n",
            "\n",
            "iter   19   |   diff: 0.12445   |   V(s0) = 6.788   V(s1) = 9.928   V(s2) = 7.680\n",
            "\n",
            "iter   20   |   diff: 0.11200   |   V(s0) = 6.912   V(s1) = 10.052   V(s2) = 7.805\n",
            "\n",
            "iter   21   |   diff: 0.10079   |   V(s0) = 7.024   V(s1) = 10.164   V(s2) = 7.917\n",
            "\n",
            "iter   22   |   diff: 0.09071   |   V(s0) = 7.125   V(s1) = 10.265   V(s2) = 8.017\n",
            "\n",
            "iter   23   |   diff: 0.08164   |   V(s0) = 7.216   V(s1) = 10.356   V(s2) = 8.108\n",
            "\n",
            "iter   24   |   diff: 0.07347   |   V(s0) = 7.297   V(s1) = 10.437   V(s2) = 8.190\n",
            "\n",
            "iter   25   |   diff: 0.06612   |   V(s0) = 7.371   V(s1) = 10.511   V(s2) = 8.263\n",
            "\n",
            "iter   26   |   diff: 0.05951   |   V(s0) = 7.437   V(s1) = 10.577   V(s2) = 8.329\n",
            "\n",
            "iter   27   |   diff: 0.05356   |   V(s0) = 7.496   V(s1) = 10.636   V(s2) = 8.389\n",
            "\n",
            "iter   28   |   diff: 0.04820   |   V(s0) = 7.550   V(s1) = 10.690   V(s2) = 8.442\n",
            "\n",
            "iter   29   |   diff: 0.04338   |   V(s0) = 7.598   V(s1) = 10.738   V(s2) = 8.491\n",
            "\n",
            "iter   30   |   diff: 0.03904   |   V(s0) = 7.641   V(s1) = 10.782   V(s2) = 8.534\n",
            "\n",
            "iter   31   |   diff: 0.03514   |   V(s0) = 7.681   V(s1) = 10.821   V(s2) = 8.573\n",
            "\n",
            "iter   32   |   diff: 0.03163   |   V(s0) = 7.716   V(s1) = 10.856   V(s2) = 8.608\n",
            "\n",
            "iter   33   |   diff: 0.02846   |   V(s0) = 7.747   V(s1) = 10.887   V(s2) = 8.640\n",
            "\n",
            "iter   34   |   diff: 0.02562   |   V(s0) = 7.776   V(s1) = 10.916   V(s2) = 8.668\n",
            "\n",
            "iter   35   |   diff: 0.02306   |   V(s0) = 7.801   V(s1) = 10.941   V(s2) = 8.694\n",
            "\n",
            "iter   36   |   diff: 0.02075   |   V(s0) = 7.824   V(s1) = 10.964   V(s2) = 8.717\n",
            "\n",
            "iter   37   |   diff: 0.01867   |   V(s0) = 7.845   V(s1) = 10.985   V(s2) = 8.738\n",
            "\n",
            "iter   38   |   diff: 0.01681   |   V(s0) = 7.864   V(s1) = 11.004   V(s2) = 8.756\n",
            "\n",
            "iter   39   |   diff: 0.01513   |   V(s0) = 7.881   V(s1) = 11.021   V(s2) = 8.773\n",
            "\n",
            "iter   40   |   diff: 0.01361   |   V(s0) = 7.896   V(s1) = 11.036   V(s2) = 8.788\n",
            "\n",
            "iter   41   |   diff: 0.01225   |   V(s0) = 7.909   V(s1) = 11.049   V(s2) = 8.802\n",
            "\n",
            "iter   42   |   diff: 0.01103   |   V(s0) = 7.922   V(s1) = 11.062   V(s2) = 8.814\n",
            "\n",
            "iter   43   |   diff: 0.00992   |   V(s0) = 7.933   V(s1) = 11.073   V(s2) = 8.825\n",
            "\n",
            "iter   44   |   diff: 0.00893   |   V(s0) = 7.943   V(s1) = 11.083   V(s2) = 8.835\n",
            "\n",
            "iter   45   |   diff: 0.00804   |   V(s0) = 7.952   V(s1) = 11.092   V(s2) = 8.844\n",
            "\n",
            "iter   46   |   diff: 0.00724   |   V(s0) = 7.960   V(s1) = 11.100   V(s2) = 8.852\n",
            "\n",
            "iter   47   |   diff: 0.00651   |   V(s0) = 7.967   V(s1) = 11.107   V(s2) = 8.859\n",
            "\n",
            "iter   48   |   diff: 0.00586   |   V(s0) = 7.973   V(s1) = 11.113   V(s2) = 8.866\n",
            "\n",
            "iter   49   |   diff: 0.00527   |   V(s0) = 7.979   V(s1) = 11.119   V(s2) = 8.872\n",
            "\n",
            "iter   50   |   diff: 0.00475   |   V(s0) = 7.984   V(s1) = 11.125   V(s2) = 8.877\n",
            "\n",
            "iter   51   |   diff: 0.00427   |   V(s0) = 7.989   V(s1) = 11.129   V(s2) = 8.882\n",
            "\n",
            "iter   52   |   diff: 0.00384   |   V(s0) = 7.993   V(s1) = 11.134   V(s2) = 8.886\n",
            "\n",
            "iter   53   |   diff: 0.00346   |   V(s0) = 7.997   V(s1) = 11.137   V(s2) = 8.890\n",
            "\n",
            "iter   54   |   diff: 0.00311   |   V(s0) = 8.001   V(s1) = 11.141   V(s2) = 8.893\n",
            "\n",
            "iter   55   |   diff: 0.00280   |   V(s0) = 8.004   V(s1) = 11.144   V(s2) = 8.896\n",
            "\n",
            "iter   56   |   diff: 0.00252   |   V(s0) = 8.007   V(s1) = 11.147   V(s2) = 8.899\n",
            "\n",
            "iter   57   |   diff: 0.00227   |   V(s0) = 8.009   V(s1) = 11.149   V(s2) = 8.902\n",
            "\n",
            "iter   58   |   diff: 0.00204   |   V(s0) = 8.011   V(s1) = 11.152   V(s2) = 8.904\n",
            "\n",
            "iter   59   |   diff: 0.00184   |   V(s0) = 8.014   V(s1) = 11.154   V(s2) = 8.906\n",
            "\n",
            "iter   60   |   diff: 0.00166   |   V(s0) = 8.015   V(s1) = 11.155   V(s2) = 8.908\n",
            "\n",
            "iter   61   |   diff: 0.00149   |   V(s0) = 8.017   V(s1) = 11.157   V(s2) = 8.909\n",
            "\n",
            "iter   62   |   diff: 0.00134   |   V(s0) = 8.019   V(s1) = 11.159   V(s2) = 8.911\n",
            "\n",
            "iter   63   |   diff: 0.00121   |   V(s0) = 8.020   V(s1) = 11.160   V(s2) = 8.912\n",
            "\n",
            "iter   64   |   diff: 0.00109   |   V(s0) = 8.021   V(s1) = 11.161   V(s2) = 8.913\n",
            "\n",
            "iter   65   |   diff: 0.00098   |   V(s0) = 8.022   V(s1) = 11.162   V(s2) = 8.915\n",
            "\n",
            "Terminated\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c9XP1eT1sYR4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 249
        },
        "outputId": "edd05678-b972-4db3-ff5f-8b84c0580b82"
      },
      "source": [
        "if has_graphviz:\n",
        "    display(plot_graph_with_state_values(mdp, state_values))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7f0eefe67b38>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n -->\n<!-- Title: MDP Pages: 1 -->\n<svg width=\"720pt\" height=\"172pt\"\n viewBox=\"0.00 0.00 720.00 172.46\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(.5807 .5807) rotate(0) translate(4 293)\">\n<title>MDP</title>\n<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-293 1235.9697,-293 1235.9697,4 -4,4\"/>\n<!-- s0 -->\n<g id=\"node1\" class=\"node\">\n<title>s0</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"75.4178\" cy=\"-114\" rx=\"71.34\" ry=\"71.34\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"75.4178\" cy=\"-114\" rx=\"75.3357\" ry=\"75.3357\"/>\n<text text-anchor=\"middle\" x=\"75.4178\" y=\"-120.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s0</text>\n<text text-anchor=\"middle\" x=\"75.4178\" y=\"-94.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">V =8.02</text>\n</g>\n<!-- s0&#45;a0 -->\n<g id=\"node2\" class=\"node\">\n<title>s0&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"264.4127\" cy=\"-158\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"264.4127\" y=\"-153\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s0&#45;&gt;s0&#45;a0 -->\n<g id=\"edge1\" class=\"edge\">\n<title>s0&#45;&gt;s0&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M150.0418,-125.5517C172.3516,-129.6738 196.7464,-134.8498 218.8356,-141 222.1916,-141.9344 225.6494,-143.009 229.0891,-144.1545\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"228.0406,-147.4958 238.6354,-147.5124 230.3634,-140.8925 228.0406,-147.4958\"/>\n</g>\n<!-- s0&#45;a1 -->\n<g id=\"node4\" class=\"node\">\n<title>s0&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"264.4127\" cy=\"-231\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"264.4127\" y=\"-226\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s0&#45;&gt;s0&#45;a1 -->\n<g id=\"edge4\" class=\"edge\">\n<title>s0&#45;&gt;s0&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M139.6666,-153.7741C170.7367,-173.0085 206.684,-195.2622 231.9855,-210.9255\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"230.4518,-214.0924 240.7967,-216.3802 234.1364,-208.1406 230.4518,-214.0924\"/>\n</g>\n<!-- s0&#45;a0&#45;&gt;s0 -->\n<g id=\"edge2\" class=\"edge\">\n<title>s0&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M237.1503,-152.6361C217.9906,-148.7832 191.7853,-143.3463 168.8356,-138 165.4299,-137.2066 161.9583,-136.383 158.4541,-135.5393\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"159.0645,-132.0857 148.52,-133.117 157.4062,-138.8864 159.0645,-132.0857\"/>\n<text text-anchor=\"middle\" x=\"193.8356\" y=\"-154.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.5</text>\n</g>\n<!-- s2 -->\n<g id=\"node3\" class=\"node\">\n<title>s2</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"539.4077\" cy=\"-205\" rx=\"71.34\" ry=\"71.34\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"539.4077\" cy=\"-205\" rx=\"75.3357\" ry=\"75.3357\"/>\n<text text-anchor=\"middle\" x=\"539.4077\" y=\"-211.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s2</text>\n<text text-anchor=\"middle\" x=\"539.4077\" y=\"-185.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">V =8.91</text>\n</g>\n<!-- s0&#45;a0&#45;&gt;s2 -->\n<g id=\"edge3\" class=\"edge\">\n<title>s0&#45;a0&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M291.7857,-162.6784C329.5283,-169.129 399.2192,-181.0401 455.0746,-190.5864\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"454.5292,-194.0439 464.9759,-192.2787 455.7085,-187.144 454.5292,-194.0439\"/>\n<text text-anchor=\"middle\" x=\"377.9899\" y=\"-194.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.5</text>\n</g>\n<!-- s2&#45;a0 -->\n<g id=\"node8\" class=\"node\">\n<title>s2&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"808.4026\" cy=\"-150\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"808.4026\" y=\"-145\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s2&#45;&gt;s2&#45;a0 -->\n<g id=\"edge13\" class=\"edge\">\n<title>s2&#45;&gt;s2&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M613.7892,-191.169C657.3227,-182.8702 713.2988,-171.8484 762.8255,-161 765.6581,-160.3795 768.5851,-159.7187 771.5251,-159.0408\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"772.5218,-162.402 781.454,-156.704 770.9181,-155.5881 772.5218,-162.402\"/>\n</g>\n<!-- s2&#45;a1 -->\n<g id=\"node9\" class=\"node\">\n<title>s2&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"808.4026\" cy=\"-77\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"808.4026\" y=\"-72\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s2&#45;&gt;s2&#45;a1 -->\n<g id=\"edge16\" class=\"edge\">\n<title>s2&#45;&gt;s2&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M589.8498,-148.3878C602.8263,-136.5386 617.4893,-125.1756 632.8255,-117 676.5157,-93.7091 733.3426,-83.927 770.333,-79.8533\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"771.079,-83.2957 780.6756,-78.8063 770.3739,-76.3313 771.079,-83.2957\"/>\n</g>\n<!-- s0&#45;a1&#45;&gt;s2 -->\n<g id=\"edge5\" class=\"edge\">\n<title>s0&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M292.0394,-228.388C329.608,-224.836 398.4925,-218.3231 454.0473,-213.0706\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"454.6288,-216.5313 464.2549,-212.1055 453.9698,-209.5624 454.6288,-216.5313\"/>\n<text text-anchor=\"middle\" x=\"377.9899\" y=\"-231.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 1</text>\n</g>\n<!-- s1 -->\n<g id=\"node5\" class=\"node\">\n<title>s1</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"1006.3976\" cy=\"-103\" rx=\"71.34\" ry=\"71.34\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"1006.3976\" cy=\"-103\" rx=\"75.3357\" ry=\"75.3357\"/>\n<text text-anchor=\"middle\" x=\"1006.3976\" y=\"-109.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s1</text>\n<text text-anchor=\"middle\" x=\"1006.3976\" y=\"-83.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">V =11.1</text>\n</g>\n<!-- s1&#45;a0 -->\n<g id=\"node6\" class=\"node\">\n<title>s1&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"1204.3925\" cy=\"-79\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"1204.3925\" y=\"-74\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s1&#45;&gt;s1&#45;a0 -->\n<g id=\"edge6\" class=\"edge\">\n<title>s1&#45;&gt;s1&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M1081.8245,-97.9414C1106.4481,-95.792 1133.8691,-92.867 1158.8154,-89 1161.6673,-88.5579 1164.608,-88.0452 1167.5577,-87.4901\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"1168.3959,-90.8916 1177.5047,-85.4805 1167.0096,-84.0302 1168.3959,-90.8916\"/>\n</g>\n<!-- s1&#45;a1 -->\n<g id=\"node7\" class=\"node\">\n<title>s1&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"1204.3925\" cy=\"-161\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"1204.3925\" y=\"-156\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s1&#45;&gt;s1&#45;a1 -->\n<g id=\"edge10\" class=\"edge\">\n<title>s1&#45;&gt;s1&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M1080.285,-119.6245C1105.4323,-125.8839 1133.5592,-133.5536 1158.8154,-142 1162.3571,-143.1844 1166.0131,-144.5159 1169.6422,-145.9102\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"1168.5033,-149.2242 1179.0879,-149.6893 1171.1036,-142.7251 1168.5033,-149.2242\"/>\n</g>\n<!-- s1&#45;a0&#45;&gt;s0 -->\n<g id=\"edge7\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1181.504,-63.203C1145.8051,-40.1488 1073.9396,0 1006.3976,0 264.4127,0 264.4127,0 264.4127,0 217.2315,0 171.715,-26.3721 137.1764,-53.7478\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"134.6289,-51.3088 129.0862,-60.3381 139.0499,-56.736 134.6289,-51.3088\"/>\n<text text-anchor=\"middle\" x=\"697.8255\" y=\"-5.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.7 &#160;reward =5</text>\n</g>\n<!-- s1&#45;a0&#45;&gt;s2 -->\n<g id=\"edge9\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1188.831,-102.2984C1184.6252,-109.1282 1180.283,-116.7204 1176.8154,-124 1166.4783,-145.7008 1177.307,-159.643 1158.8154,-175 1132.1704,-197.1281 1116.2981,-183.753 1081.8154,-187 920.9693,-202.1459 732.2512,-205.0777 625.1592,-205.3625\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"625.047,-201.8626 615.0539,-205.3821 625.0607,-208.8626 625.047,-201.8626\"/>\n<text text-anchor=\"middle\" x=\"883.4798\" y=\"-205.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.2</text>\n</g>\n<!-- s1&#45;a0&#45;&gt;s1 -->\n<g id=\"edge8\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1177.6279,-70.7289C1171.5049,-69.1802 1164.9825,-67.7983 1158.8154,-67 1132.8101,-63.6339 1125.5954,-62.2045 1099.8154,-67 1095.0975,-67.8776 1090.3107,-68.9846 1085.5247,-70.2614\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"1084.3549,-66.9563 1075.7243,-73.1014 1086.3033,-73.6797 1084.3549,-66.9563\"/>\n<text text-anchor=\"middle\" x=\"1129.3154\" y=\"-72.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.1</text>\n</g>\n<!-- s1&#45;a1&#45;&gt;s2 -->\n<g id=\"edge12\" class=\"edge\">\n<title>s1&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1192.1536,-185.8183C1184.4695,-198.4831 1173.2644,-212.7204 1158.8154,-220 984.1455,-308.0013 745.1268,-263.4284 621.2828,-230.2114\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"622.0466,-226.7919 611.4782,-227.5405 620.2067,-233.5458 622.0466,-226.7919\"/>\n<text text-anchor=\"middle\" x=\"883.4798\" y=\"-276.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.05</text>\n</g>\n<!-- s1&#45;a1&#45;&gt;s1 -->\n<g id=\"edge11\" class=\"edge\">\n<title>s1&#45;a1&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1177.0095,-156.4822C1155.7364,-152.675 1125.6126,-146.6305 1099.8154,-139 1095.2632,-137.6535 1090.6189,-136.1846 1085.9542,-134.6346\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"1086.9678,-131.2822 1076.3732,-131.3526 1084.6993,-137.9045 1086.9678,-131.2822\"/>\n<text text-anchor=\"middle\" x=\"1129.3154\" y=\"-158.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.95</text>\n</g>\n<!-- s2&#45;a0&#45;&gt;s0 -->\n<g id=\"edge14\" class=\"edge\">\n<title>s2&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M788.2368,-130.6307C780.833,-124.7608 772.0029,-119.0609 762.8255,-116 653.7772,-79.6302 319.5809,-96.8659 160.8462,-107.6243\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"160.386,-104.1476 150.6486,-108.323 160.8645,-111.1312 160.386,-104.1476\"/>\n<text text-anchor=\"middle\" x=\"377.9899\" y=\"-104.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.4</text>\n</g>\n<!-- s2&#45;a0&#45;&gt;s1 -->\n<g id=\"edge15\" class=\"edge\">\n<title>s2&#45;a0&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M835.5178,-143.5634C858.0717,-138.2096 891.4251,-130.2921 922.6929,-122.8698\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"923.8339,-126.1963 932.7551,-120.4812 922.2171,-119.3856 923.8339,-126.1963\"/>\n<text text-anchor=\"middle\" x=\"883.4798\" y=\"-144.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.6</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s0 -->\n<g id=\"edge17\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M780.7749,-74.953C705.2204,-69.749 488.8738,-57.4844 309.9899,-73 259.5414,-77.3757 203.6686,-86.9501 159.1111,-95.7468\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"158.3231,-92.335 149.2029,-97.7268 159.6949,-99.1993 158.3231,-92.335\"/>\n<text text-anchor=\"middle\" x=\"377.9899\" y=\"-78.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.3 &#160;reward =&#45;1</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s2 -->\n<g id=\"edge19\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M788.5478,-96.6791C781.0396,-103.1091 772.0641,-109.6963 762.8255,-114 709.4919,-138.8447 686.9592,-115.9508 632.8255,-139 624.147,-142.6951 615.4729,-147.2913 607.1003,-152.3268\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"605.1148,-149.4405 598.4968,-157.7141 608.8298,-155.3733 605.1148,-149.4405\"/>\n<text text-anchor=\"middle\" x=\"697.8255\" y=\"-144.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.4</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s1 -->\n<g id=\"edge18\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M835.933,-80.6152C858.0809,-83.5236 890.4476,-87.7739 921.0422,-91.7914\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"920.9619,-95.3108 931.3325,-93.1427 921.8733,-88.3704 920.9619,-95.3108\"/>\n<text text-anchor=\"middle\" x=\"883.4798\" y=\"-95.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.3</text>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.106395Z",
          "start_time": "2018-04-02T13:43:05.522Z"
        },
        "id": "0KkLFM2HsYSA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 29
        },
        "outputId": "31268cf6-f6ad-40f8-c1b0-edd6bebb9f5b"
      },
      "source": [
        "print(\"Final state values:\", state_values)\n",
        "\n",
        "assert abs(state_values['s0'] - 8.032) < 0.01\n",
        "assert abs(state_values['s1'] - 11.169) < 0.01\n",
        "assert abs(state_values['s2'] - 8.921) < 0.01"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Final state values: {'s0': 8.023123818663871, 's1': 11.163174814980803, 's2': 8.915559364985523}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9pkLlXs6sYSL",
        "colab_type": "text"
      },
      "source": [
        "Now let's use those $V^{*}(s)$ to find optimal actions in each state\n",
        "\n",
        " $$\\pi^*(s) = argmax_a \\sum_{s'} P(s' | s,a) \\cdot [ r(s,a,s') + \\gamma V_{i}(s')] = argmax_a Q_i(s,a)$$\n",
        " \n",
        "The only difference vs V(s) is that here we take not max but argmax: find action such with maximum Q(s,a)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.107338Z",
          "start_time": "2018-04-02T13:43:05.525Z"
        },
        "id": "kZ0UGznqsYSM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_optimal_action(mdp, state_values, state, gamma=0.9):\n",
        "    \"\"\" Finds optimal action using formula above. \"\"\"\n",
        "    if mdp.is_terminal(state):\n",
        "        return None\n",
        "    \n",
        "    max_value = -np.inf\n",
        "    selected_action = None\n",
        "    \n",
        "    for possible_action in mdp.get_possible_actions(state):\n",
        "        q_sa = get_action_value(mdp, state_values, state, possible_action, gamma)\n",
        "        if q_sa > max_value:\n",
        "            max_value = q_sa\n",
        "            selected_action = possible_action\n",
        "\n",
        "    return selected_action"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.108149Z",
          "start_time": "2018-04-02T13:43:05.530Z"
        },
        "id": "GyMWKon5sYST",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "assert get_optimal_action(mdp, state_values, 's0', gamma) == 'a1'\n",
        "assert get_optimal_action(mdp, state_values, 's1', gamma) == 'a0'\n",
        "assert get_optimal_action(mdp, state_values, 's2', gamma) == 'a0'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:44:05.017823Z",
          "start_time": "2018-04-02T13:44:04.962755Z"
        },
        "id": "3yBHg1mFsYSb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 246
        },
        "outputId": "d0fccc27-4a66-4bee-b51c-cdd601f2bce7"
      },
      "source": [
        "if has_graphviz:\n",
        "    try:\n",
        "        display(plot_graph_optimal_strategy_and_state_values(mdp, state_values))\n",
        "    except ImportError:\n",
        "        raise ImportError(\"Run the cell that starts with \\\"%%writefile mdp_get_action_value.py\\\"\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7f0eefde9160>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n -->\n<!-- Title: MDP Pages: 1 -->\n<svg width=\"720pt\" height=\"170pt\"\n viewBox=\"0.00 0.00 720.00 170.13\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(.5807 .5807) rotate(0) translate(4 289)\">\n<title>MDP</title>\n<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-289 1235.9697,-289 1235.9697,4 -4,4\"/>\n<!-- s0 -->\n<g id=\"node1\" class=\"node\">\n<title>s0</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"75.4178\" cy=\"-117\" rx=\"71.34\" ry=\"71.34\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"75.4178\" cy=\"-117\" rx=\"75.3357\" ry=\"75.3357\"/>\n<text text-anchor=\"middle\" x=\"75.4178\" y=\"-123.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s0</text>\n<text text-anchor=\"middle\" x=\"75.4178\" y=\"-97.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">V =8.02</text>\n</g>\n<!-- s0&#45;a0 -->\n<g id=\"node2\" class=\"node\">\n<title>s0&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"264.4127\" cy=\"-148\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"264.4127\" y=\"-143\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s0&#45;&gt;s0&#45;a0 -->\n<g id=\"edge1\" class=\"edge\">\n<title>s0&#45;&gt;s0&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M150.82,-120.6638C172.9292,-122.7295 197.0203,-125.9574 218.8356,-131 222.2297,-131.7846 225.7134,-132.7577 229.1697,-133.8382\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"228.1459,-137.1869 238.7404,-137.1004 230.4043,-130.5612 228.1459,-137.1869\"/>\n</g>\n<!-- s0&#45;a1 -->\n<g id=\"node4\" class=\"node\">\n<title>s0&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"264.4127\" cy=\"-221\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"264.4127\" y=\"-216\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s0&#45;&gt;s0&#45;a1 -->\n<g id=\"edge4\" class=\"edge\">\n<title>s0&#45;&gt;s0&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M144.1839,-148.3181C174.8806,-164.6544 209.4986,-184.1656 233.6776,-198.9495\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"232.0615,-202.0659 242.4033,-204.3676 235.7541,-196.119 232.0615,-202.0659\"/>\n</g>\n<!-- s0&#45;&gt;s0&#45;a1 -->\n<g id=\"edge20\" class=\"edge\">\n<title>s0&#45;&gt;s0&#45;a1</title>\n<path fill=\"none\" stroke=\"#00ff00\" stroke-width=\"6\" d=\"M138.6131,-158.1846C168.6814,-175.4079 203.5164,-194.2437 228.8952,-206.8652\"/>\n<polygon fill=\"#00ff00\" stroke=\"#00ff00\" stroke-width=\"6\" points=\"226.8393,-211.704 238.1291,-211.382 231.453,-202.2719 226.8393,-211.704\"/>\n</g>\n<!-- s0&#45;a0&#45;&gt;s0 -->\n<g id=\"edge2\" class=\"edge\">\n<title>s0&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M236.77,-143.4659C216.4311,-140.1298 187.6218,-135.4043 159.9627,-130.8675\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"160.277,-127.3724 149.8423,-129.2075 159.1439,-134.2801 160.277,-127.3724\"/>\n<text text-anchor=\"middle\" x=\"193.8356\" y=\"-146.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.5</text>\n</g>\n<!-- s2 -->\n<g id=\"node3\" class=\"node\">\n<title>s2</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"539.4077\" cy=\"-195\" rx=\"71.34\" ry=\"71.34\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"539.4077\" cy=\"-195\" rx=\"75.3357\" ry=\"75.3357\"/>\n<text text-anchor=\"middle\" x=\"539.4077\" y=\"-201.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s2</text>\n<text text-anchor=\"middle\" x=\"539.4077\" y=\"-175.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">V =8.91</text>\n</g>\n<!-- s0&#45;a0&#45;&gt;s2 -->\n<g id=\"edge3\" class=\"edge\">\n<title>s0&#45;a0&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M291.7857,-152.6784C329.5283,-159.129 399.2192,-171.0401 455.0746,-180.5864\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"454.5292,-184.0439 464.9759,-182.2787 455.7085,-177.144 454.5292,-184.0439\"/>\n<text text-anchor=\"middle\" x=\"377.9899\" y=\"-184.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.5</text>\n</g>\n<!-- s2&#45;a0 -->\n<g id=\"node8\" class=\"node\">\n<title>s2&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"808.4026\" cy=\"-149\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"808.4026\" y=\"-144\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s2&#45;&gt;s2&#45;a0 -->\n<g id=\"edge13\" class=\"edge\">\n<title>s2&#45;&gt;s2&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M612.5564,-176.4085C663.6493,-166.4782 729.7747,-155.5767 770.6267,-150.5966\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"771.1609,-154.0582 780.6939,-149.4352 770.3586,-147.1044 771.1609,-154.0582\"/>\n</g>\n<!-- s2&#45;&gt;s2&#45;a0 -->\n<g id=\"edge22\" class=\"edge\">\n<title>s2&#45;&gt;s2&#45;a0</title>\n<path fill=\"none\" stroke=\"#00ff00\" stroke-width=\"6\" d=\"M614.6955,-188.2566C666.1682,-180.6024 732.0962,-168.8786 772.2446,-159.9657\"/>\n<polygon fill=\"#00ff00\" stroke=\"#00ff00\" stroke-width=\"6\" points=\"773.5398,-165.0551 782.1167,-157.7066 771.1975,-154.8197 773.5398,-165.0551\"/>\n</g>\n<!-- s2&#45;a1 -->\n<g id=\"node9\" class=\"node\">\n<title>s2&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"808.4026\" cy=\"-74\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"808.4026\" y=\"-69\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s2&#45;&gt;s2&#45;a1 -->\n<g id=\"edge16\" class=\"edge\">\n<title>s2&#45;&gt;s2&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M589.7458,-138.6755C602.7265,-126.9757 617.4176,-115.8375 632.8255,-108 676.6518,-85.7068 733.4465,-77.9842 770.3924,-75.3362\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"770.9593,-78.8077 780.7217,-74.6912 770.523,-71.8213 770.9593,-78.8077\"/>\n</g>\n<!-- s0&#45;a1&#45;&gt;s2 -->\n<g id=\"edge5\" class=\"edge\">\n<title>s0&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M292.0394,-218.388C329.608,-214.836 398.4925,-208.3231 454.0473,-203.0706\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"454.6288,-206.5313 464.2549,-202.1055 453.9698,-199.5624 454.6288,-206.5313\"/>\n<text text-anchor=\"middle\" x=\"377.9899\" y=\"-221.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 1</text>\n</g>\n<!-- s1 -->\n<g id=\"node5\" class=\"node\">\n<title>s1</title>\n<ellipse fill=\"#85ff75\" stroke=\"#85ff75\" cx=\"1006.3976\" cy=\"-103\" rx=\"71.34\" ry=\"71.34\"/>\n<ellipse fill=\"none\" stroke=\"#85ff75\" cx=\"1006.3976\" cy=\"-103\" rx=\"75.3357\" ry=\"75.3357\"/>\n<text text-anchor=\"middle\" x=\"1006.3976\" y=\"-109.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">s1</text>\n<text text-anchor=\"middle\" x=\"1006.3976\" y=\"-83.8\" font-family=\"Arial\" font-size=\"24.00\" fill=\"#000000\">V =11.1</text>\n</g>\n<!-- s1&#45;a0 -->\n<g id=\"node6\" class=\"node\">\n<title>s1&#45;a0</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"1204.3925\" cy=\"-101\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"1204.3925\" y=\"-96\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a0</text>\n</g>\n<!-- s1&#45;&gt;s1&#45;a0 -->\n<g id=\"edge6\" class=\"edge\">\n<title>s1&#45;&gt;s1&#45;a0</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M1081.5333,-95.6684C1110.6936,-94.9325 1142.791,-94.9905 1166.8638,-95.9532\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"1166.8314,-99.4556 1176.9881,-96.4403 1167.1679,-92.4637 1166.8314,-99.4556\"/>\n</g>\n<!-- s1&#45;&gt;s1&#45;a0 -->\n<g id=\"edge21\" class=\"edge\">\n<title>s1&#45;&gt;s1&#45;a0</title>\n<path fill=\"none\" stroke=\"#00ff00\" stroke-width=\"6\" d=\"M1081.8158,-108.815C1110.9459,-108.9561 1142.968,-108.2471 1166.9755,-106.7983\"/>\n<polygon fill=\"#00ff00\" stroke=\"#00ff00\" stroke-width=\"6\" points=\"1167.4538,-112.0279 1177.0717,-106.1068 1166.7363,-101.5524 1167.4538,-112.0279\"/>\n</g>\n<!-- s1&#45;a1 -->\n<g id=\"node7\" class=\"node\">\n<title>s1&#45;a1</title>\n<ellipse fill=\"#ffb6c1\" stroke=\"#ffb6c1\" cx=\"1204.3925\" cy=\"-174\" rx=\"27.6545\" ry=\"27.6545\"/>\n<text text-anchor=\"middle\" x=\"1204.3925\" y=\"-169\" font-family=\"Arial\" font-size=\"20.00\" fill=\"#000000\">a1</text>\n</g>\n<!-- s1&#45;&gt;s1&#45;a1 -->\n<g id=\"edge10\" class=\"edge\">\n<title>s1&#45;&gt;s1&#45;a1</title>\n<path fill=\"none\" stroke=\"#ff0000\" stroke-width=\"2\" d=\"M1078.8475,-124.0352C1104.3931,-132.0435 1133.1359,-141.732 1158.8154,-152 1162.5631,-153.4985 1166.4334,-155.1677 1170.2616,-156.8979\"/>\n<polygon fill=\"#ff0000\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"1169.0558,-160.1978 1179.597,-161.2624 1172.0205,-153.8566 1169.0558,-160.1978\"/>\n</g>\n<!-- s1&#45;a0&#45;&gt;s0 -->\n<g id=\"edge7\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1183.9013,-82.2394C1149.7821,-53.0309 1077.9786,0 1006.3976,0 264.4127,0 264.4127,0 264.4127,0 216.3378,0 170.2964,-27.707 135.694,-56.1547\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"133.429,-53.4863 128.0434,-62.6103 137.9433,-58.8363 133.429,-53.4863\"/>\n<text text-anchor=\"middle\" x=\"697.8255\" y=\"-5.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.7 &#160;reward =5</text>\n</g>\n<!-- s1&#45;a0&#45;&gt;s2 -->\n<g id=\"edge9\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1186.3173,-122.3344C1182.8783,-126.9876 1179.5155,-132.0158 1176.8154,-137 1165.3656,-158.1348 1178.359,-174.0062 1158.8154,-188 1075.6373,-247.5578 773.9843,-222.4577 624.5282,-205.6369\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"624.8034,-202.1457 614.4716,-204.4917 624.0113,-209.1008 624.8034,-202.1457\"/>\n<text text-anchor=\"middle\" x=\"883.4798\" y=\"-230.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.2</text>\n</g>\n<!-- s1&#45;a0&#45;&gt;s1 -->\n<g id=\"edge8\" class=\"edge\">\n<title>s1&#45;a0&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1179.9971,-87.4222C1173.3281,-84.3675 1165.9638,-81.5676 1158.8154,-80 1136.2728,-75.0564 1111.6216,-76.0536 1088.8406,-79.55\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"1088.1302,-76.1205 1078.8627,-81.2552 1089.3095,-83.0205 1088.1302,-76.1205\"/>\n<text text-anchor=\"middle\" x=\"1129.3154\" y=\"-85.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.1</text>\n</g>\n<!-- s1&#45;a1&#45;&gt;s2 -->\n<g id=\"edge12\" class=\"edge\">\n<title>s1&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1188.3376,-196.5788C1180.5575,-205.6116 1170.3872,-215.0367 1158.8154,-220 943.97,-312.1488 864.2443,-253.095 632.8255,-220 629.2054,-219.4823 625.5251,-218.8717 621.821,-218.1878\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"622.478,-214.7499 611.9856,-216.2196 621.1044,-221.6139 622.478,-214.7499\"/>\n<text text-anchor=\"middle\" x=\"883.4798\" y=\"-272.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.05</text>\n</g>\n<!-- s1&#45;a1&#45;&gt;s1 -->\n<g id=\"edge11\" class=\"edge\">\n<title>s1&#45;a1&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M1176.7915,-169.5411C1155.4013,-165.6402 1125.2261,-159.1567 1099.8154,-150 1093.9867,-147.8997 1088.0521,-145.5241 1082.1432,-142.9841\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"1083.4576,-139.7382 1072.8981,-138.8731 1080.6134,-146.1343 1083.4576,-139.7382\"/>\n<text text-anchor=\"middle\" x=\"1129.3154\" y=\"-171.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.95</text>\n</g>\n<!-- s2&#45;a0&#45;&gt;s0 -->\n<g id=\"edge14\" class=\"edge\">\n<title>s2&#45;a0&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M789.4923,-128.1755C781.948,-121.3379 772.7056,-114.5785 762.8255,-111 543.0249,-31.3906 470.5685,-106.6575 236.8356,-111 212.3019,-111.4558 185.6817,-112.301 161.2674,-113.2161\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"160.8102,-109.731 150.9517,-113.6116 161.0785,-116.7258 160.8102,-109.731\"/>\n<text text-anchor=\"middle\" x=\"377.9899\" y=\"-113.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.4</text>\n</g>\n<!-- s2&#45;a0&#45;&gt;s1 -->\n<g id=\"edge15\" class=\"edge\">\n<title>s2&#45;a0&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M835.5178,-142.7003C858.0717,-137.4604 891.4251,-129.7115 922.6929,-122.447\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"923.8066,-125.7816 932.7551,-120.1093 922.2225,-118.9632 923.8066,-125.7816\"/>\n<text text-anchor=\"middle\" x=\"883.4798\" y=\"-143.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.6</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s0 -->\n<g id=\"edge17\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s0</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M780.742,-71.3149C705.1047,-64.4675 488.5683,-48.1791 309.9899,-67 259.076,-72.366 202.9597,-84.111 158.3824,-94.8698\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"157.357,-91.5173 148.4734,-97.2908 159.0184,-98.3173 157.357,-91.5173\"/>\n<text text-anchor=\"middle\" x=\"377.9899\" y=\"-72.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.3 &#160;reward =&#45;1</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s2 -->\n<g id=\"edge19\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s2</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M787.2675,-92.0463C779.924,-97.5463 771.3768,-103.1495 762.8255,-107 708.8472,-131.3051 687.5467,-113.4172 632.8255,-136 625.4073,-139.0614 617.9188,-142.6979 610.5812,-146.6397\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"608.4085,-143.8443 601.3841,-151.7758 611.8215,-149.9559 608.4085,-143.8443\"/>\n<text text-anchor=\"middle\" x=\"697.8255\" y=\"-141.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.4</text>\n</g>\n<!-- s2&#45;a1&#45;&gt;s1 -->\n<g id=\"edge18\" class=\"edge\">\n<title>s2&#45;a1&#45;&gt;s1</title>\n<path fill=\"none\" stroke=\"#0000ff\" stroke-dasharray=\"5,2\" d=\"M835.933,-78.0323C858.2731,-81.3044 891.0101,-86.0994 921.8382,-90.6147\"/>\n<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"921.3629,-94.0824 931.7646,-92.0686 922.3774,-87.1563 921.3629,-94.0824\"/>\n<text text-anchor=\"middle\" x=\"883.4798\" y=\"-94.2\" font-family=\"Arial\" font-size=\"16.00\" fill=\"#000000\">p = 0.3</text>\n</g>\n</g>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.110002Z",
          "start_time": "2018-04-02T13:43:05.538Z"
        },
        "id": "8-5Rt7nWsYSi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 29
        },
        "outputId": "f088666a-7ab7-462c-af7f-ef5ea60f3f2d"
      },
      "source": [
        "# Measure agent's average reward\n",
        "\n",
        "s = mdp.reset()\n",
        "rewards = []\n",
        "for _ in range(10000):\n",
        "    s, r, done, _ = mdp.step(get_optimal_action(mdp, state_values, s, gamma))\n",
        "    rewards.append(r)\n",
        "\n",
        "print(\"average reward: \", np.mean(rewards))\n",
        "\n",
        "assert(0.85 < np.mean(rewards) < 1.0)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "average reward:  0.93\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-gLfw2bnsYSs",
        "colab_type": "text"
      },
      "source": [
        "### Frozen lake"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.110991Z",
          "start_time": "2018-04-02T13:43:05.541Z"
        },
        "id": "BuE0Df4FsYSu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 75
        },
        "outputId": "fe581904-4e24-404b-9626-643c47c305ba"
      },
      "source": [
        "from mdp import FrozenLakeEnv\n",
        "mdp = FrozenLakeEnv(slip_chance=0)\n",
        "\n",
        "mdp.render()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "*FFF\n",
            "FHFH\n",
            "FFFH\n",
            "HFFG\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.111919Z",
          "start_time": "2018-04-02T13:43:05.545Z"
        },
        "id": "kpJF1IglsYS1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def value_iteration(mdp, state_values=None, gamma=0.9, num_iter=1000, min_difference=1e-5):\n",
        "    \"\"\" performs num_iter value iteration steps starting from state_values. Same as before but in a function \"\"\"\n",
        "    state_values = state_values or {s: 0 for s in mdp.get_all_states()}\n",
        "    for i in range(num_iter):\n",
        "\n",
        "        # Compute new state values using the functions you defined above. It must be a dict {state : new_V(state)}\n",
        "        new_state_values = {\n",
        "            _s: get_new_state_value(mdp, state_values, _s, gamma)\n",
        "            for _s in state_values.keys()\n",
        "        }\n",
        "        # <YOUR CODE HERE >\n",
        "\n",
        "        assert isinstance(new_state_values, dict)\n",
        "\n",
        "        # Compute difference\n",
        "        diff = max(abs(new_state_values[s] - state_values[s])\n",
        "                   for s in mdp.get_all_states())\n",
        "\n",
        "        print(\"iter %4i   |   diff: %6.5f   |   V(start): %.3f \" %\n",
        "              (i, diff, new_state_values[mdp._initial_state]))\n",
        "\n",
        "        state_values = new_state_values\n",
        "        if diff < min_difference:\n",
        "            break\n",
        "\n",
        "    return state_values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.112871Z",
          "start_time": "2018-04-02T13:43:05.548Z"
        },
        "id": "e2Qc88ANsYS9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 98
        },
        "outputId": "01e40bfb-c177-4f5e-f883-2c6f339cc7dd"
      },
      "source": [
        "state_values = value_iteration(mdp)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "iter    0   |   diff: 1.00000   |   V(start): 0.000 \n",
            "iter    1   |   diff: 0.90000   |   V(start): 0.000 \n",
            "iter    2   |   diff: 0.81000   |   V(start): 0.000 \n",
            "iter    3   |   diff: 0.72900   |   V(start): 0.000 \n",
            "iter    4   |   diff: 0.65610   |   V(start): 0.000 \n",
            "iter    5   |   diff: 0.59049   |   V(start): 0.590 \n",
            "iter    6   |   diff: 0.00000   |   V(start): 0.590 \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.114062Z",
          "start_time": "2018-04-02T13:43:05.552Z"
        },
        "id": "r8i3kgXHsYTM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "outputId": "9bdf114d-0fc3-4a0a-85e8-e9534fda885a"
      },
      "source": [
        "s = mdp.reset()\n",
        "mdp.render()\n",
        "for t in range(100):\n",
        "    a = get_optimal_action(mdp, state_values, s, gamma)\n",
        "    print(a, end='\\n\\n')\n",
        "    s, r, done, _ = mdp.step(a)\n",
        "    mdp.render()\n",
        "    if done:\n",
        "        break"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "*FFF\n",
            "FHFH\n",
            "FFFH\n",
            "HFFG\n",
            "\n",
            "down\n",
            "\n",
            "SFFF\n",
            "*HFH\n",
            "FFFH\n",
            "HFFG\n",
            "\n",
            "down\n",
            "\n",
            "SFFF\n",
            "FHFH\n",
            "*FFH\n",
            "HFFG\n",
            "\n",
            "right\n",
            "\n",
            "SFFF\n",
            "FHFH\n",
            "F*FH\n",
            "HFFG\n",
            "\n",
            "down\n",
            "\n",
            "SFFF\n",
            "FHFH\n",
            "FFFH\n",
            "H*FG\n",
            "\n",
            "right\n",
            "\n",
            "SFFF\n",
            "FHFH\n",
            "FFFH\n",
            "HF*G\n",
            "\n",
            "right\n",
            "\n",
            "SFFF\n",
            "FHFH\n",
            "FFFH\n",
            "HFF*\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7KlIwYYisYTV",
        "colab_type": "text"
      },
      "source": [
        "### Let's visualize!\n",
        "\n",
        "It's usually interesting to see what your algorithm actually learned under the hood. To do so, we'll plot state value functions and optimal actions at each VI step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.115092Z",
          "start_time": "2018-04-02T13:43:05.556Z"
        },
        "id": "D3gDjkHQsYTX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "\n",
        "def draw_policy(mdp, state_values):\n",
        "    plt.figure(figsize=(3, 3))\n",
        "    h, w = mdp.desc.shape\n",
        "    states = sorted(mdp.get_all_states())\n",
        "    V = np.array([state_values[s] for s in states])\n",
        "    Pi = {s: get_optimal_action(mdp, state_values, s, gamma) for s in states}\n",
        "    plt.imshow(V.reshape(w, h), cmap='gray', interpolation='none', clim=(0, 1))\n",
        "    ax = plt.gca()\n",
        "    ax.set_xticks(np.arange(h)-.5)\n",
        "    ax.set_yticks(np.arange(w)-.5)\n",
        "    ax.set_xticklabels([])\n",
        "    ax.set_yticklabels([])\n",
        "    Y, X = np.mgrid[0:4, 0:4]\n",
        "    a2uv = {'left': (-1, 0), 'down': (0, -1), 'right': (1, 0), 'up': (-1, 0)}\n",
        "    for y in range(h):\n",
        "        for x in range(w):\n",
        "            plt.text(x, y, str(mdp.desc[y, x].item()),\n",
        "                     color='g', size=12,  verticalalignment='center',\n",
        "                     horizontalalignment='center', fontweight='bold')\n",
        "            a = Pi[y, x]\n",
        "            if a is None:\n",
        "                continue\n",
        "            u, v = a2uv[a]\n",
        "            plt.arrow(x, y, u*.3, -v*.3, color='m',\n",
        "                      head_width=0.1, head_length=0.1)\n",
        "    plt.grid(color='b', lw=2, ls='-')\n",
        "    plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.116164Z",
          "start_time": "2018-04-02T13:43:05.560Z"
        },
        "id": "GPo41bC2sYTe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "ff155844-c9d6-4c6c-952b-69aae7a60a43"
      },
      "source": [
        "state_values = {s: 0 for s in mdp.get_all_states()}\n",
        "\n",
        "for i in range(10):\n",
        "    print(\"after iteration %i\" % i)\n",
        "    state_values = value_iteration(mdp, state_values, num_iter=1)\n",
        "    draw_policy(mdp, state_values)\n",
        "# please ignore iter 0 at each step"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "after iteration 0\n",
            "iter    0   |   diff: 1.00000   |   V(start): 0.000 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAALNElEQVR4nO3dbWxVhR3H8e/pvbRrgdIaKigChWY8aBiUghKZLpopW3iw0S0CIbiIQjGGYEx0El9AGuOcOnjlZIZAeDFrhJgl2mF0VjOnS0SR+TBEKA8uKnRF6bRYWnv24vb2gdbe09vzdP/8Ps2J9N577v319Gc5p9xz/o7ruohYkRd1ABE/qdBiigotpqjQYooKLaYkMz3AcZy1wNrUZyOrYEbAkUS8eBfXdZ0Lb3WG8ms7x5nnwru+xvJP+uvo9zXGiDL6aaBCa5dDTFGhxRQVWkxRocUUFVpMUaHFFBVaTFGhxRQVWkxRocUUFVpMUaHFFBVaTFGhxZSM74cORClwMzAJKABagdPAS8BXkSTqayNQMsDtTwNfhpzlh8Q9Y0T5oin07cB4oBFoBoqBycBo4lHotE/om+fbqIIMIu4ZQ8437EI7OCxkIStZyVa28imfDr5CIakynwN297o9QWA7QPnks5Sl3MANPMiDfOt1qx4ADgWT6UKllLKKVVzCJWxhi/cVQ8xYTjlrWctBDvIcz3lbKcR8MIxCp4tcQw2llJIkyWQmZy50W9dSCNQAx4ATwFGgPds0A8snn2UsYzWrSZIkQYJRjPJe6EqgvNfn+/zNBz1FXsxiHBzaaBvaE4SQMV3kucxlBCNIkPBe6BDy9Zb1KVi3cRv3cm9WL9pwVQNPLn2Sb3/Uq1jfAH8GPs/qKRno1KFaarmWa8kb4o/+5RuXc6rkVP87NmebLa1/xr3sZQxjSJAY0jOFlXEc46ijjk46h7Qdg8vXY6BTsLL+Cf0KrzCWsVRTjdP1xe9hD2/yZuaVP4IJn0zgyOQjdE7uhLnAKOBnwLPZJupvO9vppJNruIYECVxcaqmliaZB12umGYDxdeP58lCwR1i11LKe9UxkIoUU0k47G9iQcb10xtK6Ur46FNyBRxNNPMqj3M3dFHV9HOUoT/CEp3wT6ibQeqiVr8I6OHJd1/MCVS6p/4W7l2KK3XWsc+upd6sGuL/fkofLpAtuW4DLZlzuyLDuoIvbtfS/byIT3S1scV/iJbeEkszPtbErz4zh5BlaxjnMcbez3d3FLm/PFXLGPPLcm7nZfZ7n3fu5P8J8PctAHfXtrO888uikM/OT5AObgCbgC1L7zTOBIuBvwN89x7lA+uv44bOVPWdM/8qpDp8PaHI/Y/T5evi6y3EhT18kQAfwNqkDhR8DI4AW4B3gH36lGZjnjBGKe8a459N1OUKljH7SdTnEPBVaTFGhxRQVWkxRocUUFVpMUaHFFBVaTFGhxRQVWkxRocUUFVpMUaHFFBVaTMni/dDe324ajbjnA2X0ww+dgJBpNcdZ6zjOfsdx9pPhXDyRqOkN/qFSRj/pDf5ingotpqjQYooKLaao0GKKCi2mqNBiigotpqjQYooKLaao0GKKCi2mqNBiigotpoQ/pzBXBkb2vvJ8OfAb4Dvgd5Gk6k/bcUC+FTpJkg46vK8Q94GREdF2HJ5hFzo9Z28JS3iYh3mHd7ytGPJAxribxzzWs54kSe7gDu8rajv2kXWhxzCG1axmMYsBcHGZz3zOc37Q9T7mY9ppp7iymG/Kv+mZ2RHwQMYh6z0wsji4l6miinu4h8u4jEIK+Z7vmc3sjOult2NhZSHnys/13HGRbse0rAt9IzdyK7f2ue3XXR+DWc5yTnGKluktfe+I2zdiejgv8xAPUUJJ9+DNBAm2sS3jeunteG76ub53XKTbMS3rQr/AC3zBF9RQQxllJEnyOI/zKq96e4IAx335YqCDmQDcyZ2sYAXVVJNHHm20sYxl3p9A27GPYe1D/7PrYwELWMlKjnHMr1wXjRZa2M52nuVZVrCCMsqijpTTfPktR7rYkr10sWV49A8rYoquyxEqZfSTrssh5qnQYooKLaao0GKKCi2mqNBiigotpqjQYooKLaao0GKKCi2mqNBiigotpqjQYooGb0ZCGYdPgzflIqA3+IdKGf2kN/iLeSq0mKJCiykqtJiiQospKrSYokKLKSq0mKJCiykqtJiiQospKrSYokKLKSq0mKLBmwNRxuHL9cGb05jGEY70jGnLJOSBkQkSTGEKRzjifaVcGGqZCxlDNOxCz2c+NdQwlanUUstrvOZtxZAGRiZIsIhF3MVdlFLK7dzOaU57WzkXhlrmQsYQZV3oCirYxKbugZFttLGRjdzETYOu9xiP8TVfM61yGifLT/Id36XuCGC+3nVcx33cRwEFFFEEwGY2c5aznjJOrZzK8fLj8R0OChRUFtBW3tZzQ9wy5srgzSu4gklM6v5mJ0lSQAELWDDoegUUAHB4+uG+dwTwjZjFLEYxirxex74zmZlxvXTGxumNfe+IW1mAtultfW+IW8ZcGbz5Bm/wIR+ymtUsYhEJEjzCI7EavPkUT/EyL7OOdcxiFkmSrGIVpzjl7QniPtQSqK2rZc+hPRzkYNRRBpZLgzebaWYrW9nNbpaylPd5369cvjnKUR7gASqoYCELOcOZqCNJgHz5LUczzexilx9PFZijXR9im/5hRWxxXdfzAlUuqQs3xHBxu5aoc4SbsYEGdzazY50xqGWgjuontJiiQospKrSYokKLKSq0mKJC56iRjGQc4wAoo4xSSiNOFA+6nG6o/Mu4jW3MZCb55NNBB+20s4xldNAxzGfOhe2YosvpGtJAQ3d5HRwOcMCHMuc+FTpH1VNPO+0AtNPOMzwTcaJ4UKFzVDvt7GQnnXTyHu9xnONRR4qF8M8pFN/UU08VVexgR9RRYkMHhaFSRj/poFDMU6HFFA3ejIQyDp8Gb8pFQAeFoUplHMImD53TvfnivB1TdFAo5qnQYooKLaao0GKKCi2mqNBiigotpqjQYooKLaao0GKKCi2mqNBiigotpoR7TmFEs+uGLO4zAIHybeWcOHui3+0H1h1gzvg5ESQawATgp8BEoBBoBU4D+4F/B/OSOkl2MDkwA3DJtCVUlFZ0f15WVBZhml6uBH5Fah+gCTgMFJAq+SxU6EgMcQbgWMZyPdezj3200hpYrN7WVK6heka158efffss546e49Lll5KXDGiPcwSwhFSZPwBegO55rA4wNpiXhagKHfLsuqz1zgkZR6YtZCHrWc8a1lBHHXvZG3ixdxzYwevHX+/+fNsvtg36+JOPnuTMX8/Q+GAjUx+bGkyxJ0LXWEh4A/oMF3YJ9MSnaAod8uy6rF2Qs2FfQ8ZVOuigiCJWspIVrGATmwKdDvbi4Rf7fF79S28/rc9/fp7DNYdp/G0j896dR/64fP9Cjez156+7/vtzUvvTaZv9e7neoil0yLPrslVWV0btoVrySX2zj3Fs0MdPYUr3INJOOmmmmRZaAs24c+JOrvzDlT03XDX441s/6vobwwE6oWhGEU6+z6db9T7WKAbOACeBfwE/8felLqR96EE00UQNNZ4ffwu3sIENfMZnPM3TvMVbAaZLKbm2hKvvvNrz4z+45QOaX2ym5IYSKn5fwei5o/0P9Rmp32gUAdcBfyF1UNiCCp1L9rGPE5yI5QDStGl/nMb5zecZXRlAkdPagXrgVlLHIZcB/wHGBPeSaSq0j9poi3WZAQouL6Dg8oLgX+hDUj+RF5I6SJxDalfkCPBRcC+ryxiESpcx8JMuYyDmqdBiigotpqjQYooKLaao0GKKCi2mqNBiigotpqjQYooKLaao0GKKCi2mqNBiigotpgzx/dDO/0hdrSKuxgL/jTpEBsroj8mu6/a7CMlQz1j5xHXdeT4F8p3jOPvjnA+UMWja5RBTVGgxZaiF/lMgKfwT93ygjIEa0kGhSNxpl0NMUaHFFBVaTFGhxRQVWkz5PxH5ycA1hCP1AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "after iteration 1\n",
            "iter    0   |   diff: 0.90000   |   V(start): 0.000 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAALtUlEQVR4nO3dfWwUdR7H8fd0t91rsd0Wy2Mt7IECPp2AgAQ0F43iGUAJencUDV6OiDUxhou5cHLkAvaM59PhH6ceRiNnDJSIURP0fEajBhPxEFGPIpQThQKlBRGKbWnn/thuH5fuQ2d2Zn98Xs0G9mF2P7t8WGaWnflatm0jYoocrwOIOEmFFqOo0GIUFVqMokKLUYKJbmBZ1hJgSfTcoMthgsuRRJLxGbZtW70vtVL52M6yptjwmaOxnBN7Hn2eo48oo5PiFVqrHGIUFVqMokKLUVRoMYoKLUZRocUoKrQYRYUWo6jQYhQVWoyiQotRVGgxigotRlGhxSgJvw/tihJgFjAKCAFNwGHgNeCoJ4l6WgoUx7n8n8DBDGc5E79n9CifN4X+LTAcqAUagCJgNFCIPwodU0PPPCe9CtIPv2fMcL4BF9rCYiYzWchCVrOab/im/wXyiZb5FPB8t8sDuLYClEcec5nL1VzNMpZxMtlXdRuw051MvZVQwm3cxmAGs4pVyS+YwYwRIixhCdvZzgY2JLdQBvPBAAodK3IllZRQQpAgoxmduNDNHad8oBLYC3wL7AFa000TXx553MiNLGIRQYIECHAO5yRf6ElApNv5N5zNB11Fns1sLCyaaU7tDjKQMVbkyUwml1wCBJIvdAbydZf2Llg3czN3c3daD7r54s08NvcxTv6sW7FOAOuAA2ndJfF2HaqiihnMICfFt/4FSxdwqPhQ3ytWppstpm/Gl3iJMGECBFK6p0xlHMYwqqmmnfaUXkf38nWJtwtW2u/Qb/M2pZQyj3lYHU9+Ixv5iI8SL/wVlNWUsXv0btpHt8Nk4Bzgl8D6dBP1tYY1tNPOFVxBgAA2NlVUUU99v8s10ADA8OrhHNzp7hZWFVXcxV2UU04++bTSyj3ck3C5WMaS6hKO7nRvw6Oeeh7kQe7gDgo6fvawh0d5NKl8ZdVlNO1s4mimNo5s2076BJfbRP8Kd56KKLLv5E77dV63L49zfZ9TDjajel02HZuV2NyeYNl+T3bHqe915ZTbq1hlv8ZrdjHFie9raUeeCQPJk1rGiUy017DGXsva5O4rwxlzyLFnMct+kRfte7nXw3xdp3gddWyv7xxyaKc98Z3kAcuBeqCO6HrzhUAB8C7wYdJxeok9jzPvrZx0xthHTtU4vEGT/Rm9z9fF0VWO3pJ6kgCngS1ENxQuAHKB48CnwMdOpYkv6Ywe8ntGv+fTcTkyShmdpONyiPFUaDGKCi1GUaHFKCq0GEWFFqOo0GIUFVqMokKLUVRoMYoKLUZRocUoKrQYRYUWo6Txfejkv27qDb/nA2V0wpl2QEi0mGUtsSxrq2VZW0mwL56I1/QF/4xSRifpC/5iPBVajKJCi1FUaDGKCi1GUaHFKCq0GEWFFqOo0GIUFVqMokKLUVRoMYoKLUZRocUomZ9TmC0DI7sfeT4C/A74CfibJ6n60usYl2OFDhLkNKeTX8DvAyOzhV7HHgZc6NicvTnMYQUr+JRPk1swwwMZjaXXsYe0Cx0mzCIWMZvZANjYTGUqLbT0u9zXfE0rrRRNKuJE5ETXzA6XBzKmrPvAyCIPcyRgTbKwI932OjrLX8e0C30N1zCf+T0u+3XHT38WsIBDHOL4+OM9r/DbH8R4rwMkxx7faxe6s/x1TLvQL/MyddRRSSVDGEKQII/wCO/wTnJ34OK4L0fE25jxoarqKjbu3Mh2tnsdJb4Mv44DWof+pONnOtNZyEL2stepXCJpceRTjlixRbym/1gRswx01rd/Tmeeo+2fk/MZN7PZvozLfJ3RrVO8juodWoyiQotRVGgxigotRlGhxSgqtBhFh9PNKOcyrmAF05hGIYW00cZ+9rOYxal9hTeubHgdo3Q4XYPUU0+IEAABApzu+DnbqdBZaj3rO796e4pTPMVTHifyBxU6Sx3nOK/wCm20UUcdW9nqdSRfUKGz2HrWc4xjPMmTXkfxDW0UZpQyOkkbhWI8FVqMosGbnlDGgdPgTTkLaKMwo6IZDxyo8zjHmY0cOaLjd35+HaO0USjGU6HFKCq0GEWFFqOo0GIUFVqMokKLUVRoMYoKLUZRocUoKrQYRYUWo6jQYhQVWoyiwZvxZEHGaeum8f2J7/tc/tb8t7ik9BIPEvWS7YM3xzGO3ezuGtOWSDYMjMyCjNeOupZIUaTz/Ln553oXxgcGXOipTKWSSsYwhiqqeI/3klswGwZGZkHGigkV3BC5wesYvpF2occyluUsZwQjyCefZppZylKu47p+l3uIhzjGMcZNGse+yD5+4qfoFX6brwfkTcrjdOS0f4eDAuv+s44tB7Z0nr9/xv0epokjWwZvnsd5jGJU5x92kCAhQkxner/LxY7Htmv8rp5X+LAsLeN7TcX1YcZ3j7wLR7rO+67Q2TJ48wM+4Eu+ZBGLuJ7rCRDgAR4wZ/Am0aGWwZ1B7uM+r6OcUVV1FRV/raBgRoHXUeLLpsGbDTSwmtU8z/PMZS6f87lTuUTS4sinHA00sJa1TtyVyIDoP1bEKDouRz82s5lP+MTBdWjnj8tRM7KG8o3ljq1D67gcIj6iQotRVGgxigotRlGhxSgqdBy55DKMYQCECTOUoR4n6qvteBut37UC0FrXyul6jXQDFTquCip4gRcAuJAL2cAGzud8j1P1tP/2/dReVQvAwT8cpHZ6LXaL3w9S7j4VOo6P+bjHEMujHGUvez1M1FfhTYVYwY6PYduh4MoCrDz/f3bsNhU6jj3sYQc7aKONJpp4hmdoo83rWD2EK8JYoWiBrTyLIcuHeJzIH1ToM3iap2mnnWaaeZM3vY7TR04oh9I/lkIOFFxVQGh8yOtIvpD5fQqzxG528yqv8gVf+O7dOSZcEabpwyZKl5V6HcU39F2OjNKMFSfpuxxiPBVajKLBmx7o+mfdz/z+OmrwppwFjNsozIYNrhRe8oyzOt/4tFEo4jkVWoyiQotRVGgxigotRlGhxSgqtBhFhRajqNBiFBVajKJCi1FUaDGKCi1Gyew+hR7NrkuV72cAApHHI3z7w7d9Lt925zYmDp/oQaI4yoArgXIgH2gCDgNbgf+685DaSbYf2TADcM64OYwtGdt5fkiBTw5ncBFwC9F1gHpgFxAiWvJLUaG9kOoMwNa6Vn7c9CPhBWEChQEXk3VZPGkx8ybMS/r2P2z5gVN7TjF0wVBygi6tceYCc4iWeQfwMnTOY7UAF3dS96bQGZ5dl671O9enNAPwxJsnqL+/niMPH2HwXYMpuaPE9WI/u+1Z3v/f+53nH//V4/3eft+D+2j8dyO1y2oZ89AYd4pdDsQGCnwAPYYL27i645M3hc7w7Lp0vbOv54i6W2+5NfFCAbBP2jT+o5HGJxop+1cZg64c5FJC2LRrU4/z825I7t265UALuyp3UfunWqZ8NoW8YXnOher+dI91/Hot0fXpmJXOPVx33hQ6w7Pr0rXm4jVcWnUpdnPHPlMJ/iK21LRE/5ltAwIQHBokMNjdd+jnyp/jor9f1HXBxf3fvumrpuhvLKLHxJvgwjHxus9ELwIagX3AF8AvnH2o3rQO3Y9gWZDIG5Gkb3907VEO//kwuT/PZehfhjJo1iAsy91984pnFDPt99OSvv2Om3bQsKmB4quLGfvwWAonFzof6juin2gUAFcBrxLdKDyOCp1Nwr8JE7ogRP6MfNeLnK5xT42jZWULhZNcKHJMK/A6MJ/o9tII4Hsg7N5DxqjQDsopyKFgpk9HFHcIjQwRGpmBAzt+SfQdeSbRjcSJRFdFdgNfufewOoxBBukwBs7SYQzEeCq0GEWFFqOo0GIUFVqMokKLUVRoMYoKLUZRocUoKrQYRYUWo6jQYhQVWoyiQotRVGgxSorfh7Z+BGrcizNgpcARr0MkoIzOGG3bdp+DkKS6x0qNbdtTHArkOMuytvo5Hyij27TKIUZRocUoqRb6aVdSOMfv+UAZXZXSRqGI32mVQ4yiQotRVGgxigotRlGhxSj/B2vv+jQcb73vAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "after iteration 2\n",
            "iter    0   |   diff: 0.81000   |   V(start): 0.000 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAMXUlEQVR4nO3df2zU9R3H8ef3eu3RFlqqpSAVWyUCQ5gUKxJQJ4b4E6RB2SgaNn+ALHGZzDjMZgaxM24Dp3+4zApGt0SoCQlZwjZEBBdcmAHWMYYCQlV+FWiBWqBAS++7P+5aWnq01+v3e9/vfXw9mob7Xu/be/fLiy+f77ff7+dt2baNiCkCXhcg4iQFWoyiQItRFGgxigItRgn29ALLsuYD8yNL2bfAKJdLEonHdmzbti5/1urNaTvLKrVhu6NlOaft5+jyM/qIanRSrEBryCFGUaDFKAq0GEWBFqMo0GIUBVqMokCLURRoMYoCLUZRoMUoCrQYRYEWoyjQYhQFWozS4/XQrsgD7gGuA0JAE3Ac+CtwypOKOnsWGBjj+TeBo0mu5Ur8XqNH9XkT6B8AQ4Aa4ASQAxQBA/BHoNvsoXM9Z70qpBt+rzHJ9fU50BYWk5nMHObwGq/xBV90v0ImkTCfA/7c4fk0XBsAZZDBdKYzhSksYhFn492q1cBud2pyTBJrLKaY+cxnBzt4n/fjWynJ2zDhQLcFeQELyCOPIEGKKOo50Bein5nAAuBL4GtgP9CSaDWxZZDBQzzEXOYSJEgaafSnf/yBLgGKOyyvc7Y+RyShxrYgj2c86aSTRlr8gU7yNkz4FqyHeZhneCahN9100yZenf4qZ/t1CNYZYCVwJKFvSaxbhyqoYBKTCPRy1z/72dkcG3is6xeWJFpbGwdvb7rSGHVJX79x5xoHM5gqqggT7tV2dG8bXhLrFqyE99Af8iH55FNGGVb0h1/Naj7hk55X3gWFewrZV7SPcFEYxgP9ge8BqxKtqKtKKgkT5jZuI400bGwqqKCOum7XO8EJAIZUDeHobj8cYV1ZRVUFq3evZgc7XPn+ddTxCq8wj3lkRT/2s59lLOt2vbZtWFhVSNPuJk4l6eAo4UA30kgllaxiFeWUM4MZVFPN7p4GTAHgWuAAkWHGfiJnOe4DMhKtJrZDHGIxixnGMJ7iKUopZSc7aaAhrvWP+uJ0gbfChFnPejawgalMZR7z+JzPe/57jjrMYZcr7KzPB4VtwV7OcsKE43vHJ4A6oJbIuPk70a/V9LWa2A5ykMUsJkAgvhqli47B9vM2dOy0Xdw/5EVgC5EDhRuBdKAR2Ar806lqYvPzX0Sq8P02tG077k+4xSZy1ODDTzv66XUdya1xE5vsm7nZ1zW69Rkro/rVtxhFgRajKNBiFAVajKJAi1EUaDGKN5ePSp/NYhallAKwkIXsYhfLWIaN7XFl3lKgU9QYxrQHuogicsghQIBWWj2uzFsacqSoFaygJXq9bRNNrGDFtz7MoECnrIMc5FM+JUyYC1zgAz7wuiRfUKBT2ApWtP+pvXOEeqwklfM1FlBAPfUOXjSUCtsxwtEL/MUfjnPc6xJ8RUMOMUoCe2i/n+f0e32gGp0Qe0jU4x7asqz5lmVtsyxrGz3ciyfiNR0UJlWkxurq/3hcx5WVlIyLPvLzdoxQ400xngItRlGgxSgKtBhFgRajKNBiFAVajKJAi1EUaDGKAi1GUaDFKAq0GEWBFqMo0GKU5N+ClSoNI6u41I6sGPgRcB74jSdVdfHA+geoPVfb5fmqu6oYmTvSg4ou49F2dCzQQYJc5GL8K/i9YWSKuHPwnVybfW37cl5GnofVeK/Pgc4jj8d4jGlM40VeZCtb41sxFZpapoCyojKmXDPF6zJ8I+FA55LLXObyIA8CYGNzK7fSTHO3633GZ7TQQk5JDmeKz1y6/d5vTS07NozM8bCOHqz5ag3b6re1Lz8/9nkPq4khydsx4UDfzd3MZGan52ZFP7ozm9kc4xiNIxs7f8FvgfbBMDQem49vpuNMBr4LdJK3Y8KBXsMaaqllAQsYxCCCBFnKUjawIb5v0PFgwY9iHcz4UEVVBfc+dy/ppelelxJbkrdjn8bQ/4p+TGQic5jDl3zpVF0iCXHkLEdbsEW8pl+siFE0L0dSOT8vR0NJA9nLsx0bQ2teDhEfUaDFKAq0GEWBFqMo0GIUBVqMokCnqLMvnKXhzobI46fP0ljWiN3i90nK3adAp6hAQSByoTxAGEhHHXNQoFNW6PEQpEUXMiHzZ5lYlv9/GeI2BTpFBfIChGaFIACBwgDBido9gwKd0kKPh7DyLDKf0965jf5Zp7BAXoDcDblel+Er2kOLURRoMUoCl49u6/mFIq6zErt8VI03JZUYd4F/KjS1PHKk64xHfjF06DXRR/4/a6IL/MV4CrQYRYEWoyjQYhQFWoyiQItRFGgxigItRlGgxSgKtBhFgRajKNBiFAVajKJAi1HUeDMG3ze1BCasnMChM4e6PL9+5nrG5I/xoKLLpHrjzV5LoPHmCEawj32XWsG5LBWaWk69birFOcXty1dnXu1dMT7gXaB72Xjzeq6nkkrqqOMt3mIjG10Pdio0tSwfVc79xfd7XYZveBboQEmAkuISQoQAmL5uerevH8xgwoQZxCAWspD5zGcpS+PvXJuANXvXsPXYVqy0yI0RvusBCKz890q2HNnSvvzSpJc8rCaGVGm82VfhkWG2d7id6+V1L8e9bkb0YzSjXQ305obN0HBp2Y+B/qj+I6i/tOy7QKdK480+u6zx5hS6/699OMN5kzdppZWtbGU5yznAAVdLrKiq4K6r76L/G/1dfZ++qKiqoPzX5WRNyvK6lNhSqfFmMh3mMO/xHhvZ6HqQJXWlTKDPc553edfrMsTn9IsVMYrm5ehGQ0kDwclBx8bQbszLsWfoHoatHubYGFrzcoj4iAItRlGgxSgKtBhFgRajKNAx2BdsWo+0Rh432IRrk3N1X2+0NrbScrAFgJbaFi7WXfS4In9QoGM4/855Tj90GoDWXa00PtDIxd3+CszhHx6m5o4aAI4uPErNxBrsZjXeVKBjSJ+S3ul3qFaeRdrwtCuv4IEBMwZgBaOnYcOQdXsWVob/zx27TYGOITgySLAkGNk6mdDvJ/2w0v0VltzyXKxQpCYrw2LQLwZ5XJE/KNBX0O+n/SAAVj+LjGkZXpfTRSAUIP/5fAhA1h1ZhEaGvC7JF1Lm4qRkC44KkjErg+D4oO/2zm1yy3Np2txE/qJ8r0vxDQW6G1k/9+k1xlGBUIDCtwu9LsNXNOQQoyjQYhQ13pQUpcab8i1g3AX+qdDUshebPOms9n2eP8/sdKQL/MV4CrQYRYEWoyjQYhQFWoyiQItRFGgxigItRlGgxSgKtBhFgRajKNBiFAVajJLcW7A86l3XW77vAQgUv17M19983eX56qerGTdknAcVxVAI3A4MAzKBJuA4kUvqP3fnLXVPYTdSoQfgtBHTGJ43vH15UJZPpjMYDTxCZAxQB+wFQkRCPhYF2gu97QHYUtvC6bWnyZ2dS9qA5ExM82TJk5SNKov79d9s+YZz+89RMLuAQNClEWc6MI1ImHcCa6C9paQFuHiTujeBTnLvukSt2r2qVz0Az3xwhrqX6qj/XT1X/fgq8ubluR7st6vf5uOvPm5ffv2+17t9/YFXDnDy7yepWVTDDb+9wZ1gDwPabpj/B3Tqj2rj6o1P3gTaH+2ye7ThwIZOy48+8mjPK6WBfdbm5BsnOfmHkxT+qZDs27NdqhDW7l3babns/vj21s1Hmtm7YC81L9RQur2UjMEOTqbT8cdt6/M4lch4us0S596uI28CneTedYmqvKmSsRVjsS9E75nq4R9i857myH+zrUAaBAuCpF3l7h76nWHvMPr3oy89cVP3r2/a1RR5YBGZE2+UC3PidezbngOcBA4A/wW+6+xbXU5j6G4EC4MUryuO+/Wn3j3F8V8eJ/36dAp+VUD2PdlYlrv35g2cNJAJT0yI+/U7Z+zkxNoTDJwykOG/G86A8QOcL+ogkTMaWcAdwF+IHBQ2okCnktzv5xK6MUTmpEzXg5yoEX8cQfOSZgaUuBDkNi3A34CZRI6XrgEOAbnuvWUbBdpBgawAWZP9PX1YaGiI0NAkTOz4PyJ75MlEDhLHERmK7AN2ufe2msYgiTSNgbM0jYEYT4EWoyjQYhQFWoyiQItRFGgxigItRlGgxSgKtBhFgRajKNBiFAVajKJAi1EUaDGKAi1G6eX10NZpYI975fRZPlDvdRE9UI3OKLJtu8skJL29Y2WPbdulDhXkOMuytvm5PlCNbtOQQ4yiQItRehvot1ypwjl+rw9Uo6t6dVAo4ncacohRFGgxigItRlGgxSgKtBjl/1yaMBJ8ib/pAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "after iteration 3\n",
            "iter    0   |   diff: 0.72900   |   V(start): 0.000 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAMxUlEQVR4nO3de2xUZR7G8e+ZTju9iKUCRVuFYuXihZWbQEBdiYBgcCGiu6AxrhIQImZdjavZTRZiN2HX6EqMBl2jLpJAFROisooVgxs0RgVZZZGL0kUUkEJLLTdpO3P2j9MbdGhn2jmXeXk+xjjvccbzO8eH03dOz/u+lm3biJgi5HcBIqmkQItRFGgxigItRlGgxSjhzt5gWdY8YJ7TyhsJQ1wuSSQRm7Ft2zpzq5XMbTvLGmXD5pSWlTrNx9HuGAPEqbGi4n2f6zi7yZMnNb0K8nl0xAu0uhxiFAVajKJAi1EUaDGKAi1GUaDFKAq0GEWBFqMo0GIUBVqMokCLURRoMYoCLUZRoMUonT4P7YoCYDLQD4gAJ4Aq4F/AEV8qOt2DQM84258HfvS4lrO4a/NdHDx1sN32ZVcvozSv1IeKzuDTOfQn0L8BLgQqgWrgfKA/0INgBLrZTk6v57hfhZzdmIIxFGUXtbTzw/k+VhOHx+ew24G2sBjPeO7gDp7mab7hm44/kIMT5pPAq222ZxC8DtAWYEfib7+Kq1jAApaznM/4zLWy2ppSOIXxvcZ7sq8SSpjHPL7kS17jtcQ+lOQ57K4uB7o5yPOZTwEFhAnTn/6dB/pU0985wHzgf8B3wG6goavVuGQ4UNKmva7jt5dSyiAGsYhFVFHFMpa5Hux1Vev4qu6rlvaCAQtSvo/mII9gBJlkkkFG4oFO8hx2V5eHYM1kJgtZ2KWdbrhyA0/d8hTHs9v8/DkGrAT2d+k/SUqHYJ2l/7dh8YZOPxojRqjNj5olLKGCiqZW6oZgna0PXTGuIs67E3fmEKy+9KWc8nbH1ZlZD87iYM/29bG4W+WdJt4QrC5fod/nfXrTmxnMwGo6+Dd4g4/4qPMPb4PincV82/9bYv1jMAI4D/glsKqrFaVeuDzMZTsua2kvoOOr3yQmcQu3UE89NjarWc3HfOxqjWXlZYydMxb7anemdDvEIZawhLnMJbfpr93s5kme7PBz1VQDUFxezIkdJzji0ZejLge6jjpe4AVWsYrZzGY609nCFnZ01mEKARcDe3G6Gbtx7nJMAbK6Wo07Gmns/Hja6EMfpjKV1azmdV7neBC/RSYpRowKKljPeiYykbnMZTvbEz4v+9jncoWn6/aXwuZgv8iLxIgltsd7gUPAAZx+8+VN/66yu9X4ayMb+ZiPEzsPaaZtsIN8fCm7bZfwQTYCn+B8URgIZAJ1wOfg8k9nTwT5f3YqBP34vL8PHQPe83yvyVnqdwGdWzFyBQCZizNppNHnauLw6RwG7c6vSLco0GIUBVqMokCLURRoMYoCLUbx5/FR6bbQGyGszc4jBxnPZBC7PEbsodg5f4lSoNOUtc3C+sIJtPW9RaguRCymQJ/jh5++ovdEnd+yAna2TfTeqC5PKNDpqx/Y19jYlg0RsCdpAVVQoNNa9J6o809dnVvoNKSzftC4ohF6+V1IcCjQ6a7Q7wKCRV0OMUoXxhRucrEckURZXVvWzbKseZZlbbIsa5MzzEQkuLTwpqecGrds+Y/PdZzd8OHDml4F+Tw6tPCmGE+BFqMo0GIUBVqMokCLURRoMYoCLUZRoMUoCrQYRYEWoyjQYhQFWoyiQItRFGgxivdDsIK+qGVzfeW0LkdWAvwW+Bn4qy9VtXNzxc0cOHmg3fbyG8oZnD/Yh4rO4NN59G9MYRosapkOru97PRfnXdzSLsgq8LEa//kXaI8XZDTVjP4zmHDRBL/LCAz/Aj0crBILu3mkicsLMiat7YKR5/tYRyfW7FnDpsOt4zwfGfqIj9XE4fF59C/Qg2kNMwQv0AHohiZiY9VGqGptBy7QHp9H3wK9sHwhM3fMZAIB/XEZ78tMAJWVl3HTwzeROSrT71Li8/g86radGEWBFqMo0GIU3+bleJmXGcCAFPahz815OWqH15L3Yl7K+tCal0MkQBRoMYoCLUZRoMUoCrQYRYEWo3ge6IEM5E3eZAADAHiLt7iRG70uI+0df+w4tdfXOq/vO07djDrsBq2E5Xmga6ghh5yWdhZZVLV9ukYSEioMOQ/KA8Rw1izUijneB7qaat7lXeqpJ0aM3exmK1u9LiPtRe6JQEZTIwdyHsrBsoL/yxC3+dKHfpVXsbGpp57ned6PEtJeqCBE5PYIhCBUHCI8Vpdn8CnQ1VTzNm/zNV/r6twNkXsiWAUWOQ/r6tzMtz/Wz/GcX7s2RqggRP76fL/LCBTdthOjKNBiFC28KWlKC2/KOSDpK3RFxRIXy+m6yZMnAemxqOX+/e1nPAqKoqKLml4F/66JHvAX4ynQYhQFWoyiQItRFGgxigItRlGgxSgKtBhFgRajKNBiFAVajKJAi1EUaDGKAi1G8XxM4V2b7+LgqYPtti+7ehmleaVelxNX4Be1BEavHM0Px35ot73i1gqu6n2VDxWd4VxbeHNMwRiKsota2vnhTgZ72sA3QCmt81G4LB0WtZzYbyIl55e0tHvl9PKvmADwLdBTCqcwvtf4xD+wBzIXZmL3sonOiWJPsF0Pdjosajl7yGymlkz1u4zA8C3Q6w6sY+uGrXDKad//9f0dvt+qsrAtG6vaIuOZDHgJog9Fsa9xbz63NbvW8PnBz7EynIERgVsDEFj5xUo+2f9JS/vxcY/7WE0c58rCm5/WfQqtPQ4eWP5A4h+uBxrA2m65GuiNtRuhtrUdxEB/cPgDONzaDlygz5WFNxcNXnRal6OhoqHjD+yG8MIwZIA90ul20M/dGsvKy7ih1w2c9+x57u6oG8rKy5j9l9nkjsv1u5T4PF54M30mRCuC2KwYsQkx14Ms6St9Ap0DsbtjflchAadfrIhRPL9Crxi5wutdJu2dye8AULu4FpK4s+ilz+74DICdi3f6XMlZLI2zbQ+w2N3d6gotRlGgxSgKtBhFgRajKNBiFAU6DvuUTXR/1HldaxM7ELz739G6KA3fO79dbTjQQOOhRp8rCgYFOo6fX/mZo786CkB0W5S6m+to3BGswOy7ex+V11UC8OPvf6RybCV2vRbeVKDjyJyQedodeqvAIqPUo4ewE9Rjeg+scNP0yDHIvTYXKyv4czq7TYGOIzw4THh42Dk7OZD9QDZWZrDCkj87Hyvi1GRlWfT5Yx+fKwoGBfossn+XDSGwsi2ypmX5XU47oUiI3o/0hhDkXpdLZHDE75ICIX0eTvJYeEiYrNuzCI8IB+7q3Cx/dj4nNp6g96O9/S4lMBToDuT+IaDPGDcJRUIUv1TsdxmBoi6HGEWBFqNo4U1JU1p4U84BXbhCb3axnO5wjiMdFrVM4pR7zmq55gXzzk5bWnhTjKdAi1EUaDGKAi1GUaDFKAq0GEWBFqMo0GIUBVqMokCLURRoMYoCLUZRoMUo3g7B8mntumQFfg1AoGRpCd/99F277Vvu28KwC4f5UFEcxcC1wCVADnACqMJ5pH67O7vUmMIOpMMagNMGTaO0oHXB0j65AZnO4ArgNpw+wCFgFxDBCflQFGg/JLsGYMOBBo6uPUr+rHwyengzMc2c4XOYMWRGwu//6ZOfOLn7JIWzCgmFXepxZgLTcMK8FVgDNM+mZgEuDlL3J9Aer13XVat2rEpqDcBj7x3j0OOHOPzEYS5YcAEFcwtcD/ZLW17iwz0ftrSXTok3dX6rvUv2UvNuDZWPVnLp3y51J9iXAM0D5v9Na5jBGYfh4sAnfwIdjOWyO7V+7/rT2nfedmfnH8oA+7hNzbM11DxXQ/HyYvKuzXOpQli7a+1p7RlTE7ta1++vZ9f8XVQ+VsmozaPI6pvCyXTaHm7zOo8TcfrTzRanbndt+RNoj9eu66oXrnyBoWVDsU81jZnq5A9i/c5658dsFMiAcGGYjAvcvUK/cskrXPH3K1o3XNnx+09sO+G8sHDmxBviwpx4x9u8Ph+oAfYCXwG/SO2uzqQ+dAfCxWFK1pUk/P4j/zxC1Z+qyByQSeGfC8mbnIdluTs2r+e4noy+d3TC7986fSvVa6vpOaEnpU+U0mNEj9QX9T3OHY1c4DrgTZwvhXUo0Okk/9f5RAZGyBmX43qQu2rQskHUL66nx3AXgtysAXgHuBXn+9JFwA9Avnu7bKZAp1AoN0Tu+GBPHxYpihAp8mBix//iXJHH43xJHIbTFfkW2ObebjWNgYc0jUFqaRoDMZ4CLUZRoMUoCrQYRYEWoyjQYhQFWoyiQItRFGgxigItRlGgxSgKtBhFgRajKNBiFAVajJLk89DWUWCne+V0W2/gsN9FdEI1pkZ/27bbTUKS7IiVnbZtj0pRQSlnWdamINcHqtFt6nKIURRoMUqygf6HK1WkTtDrA9XoqqS+FIoEnbocYhQFWoyiQItRFGgxigItRvk/uc1fRjPAjMcAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "after iteration 4\n",
            "iter    0   |   diff: 0.65610   |   V(start): 0.000 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAANTklEQVR4nO3dfWxd9X3H8fc5915fP+basbkEBxKnLiQLZZBHWMJDokEaMtpGiG54DGWQjYLaqB0dpANpWHgSUE0Taosi0BDrUsX+o1K0idE0pW2qDKGUPExkSUlGDCSQECcxjh0/Xt979sfxQxw7tq99z8P95fOSIvl3OFf368PHx797fM7vazmOg4gp7KALEMklBVqMokCLURRoMYoCLUaJTrSDZVmPAY+5o5IlsMDjkkQmVlp6hI6ODuvS7VY2l+0sa6kD+3JaWO6430djY1PAdVxeXd2DAOzc+auAK7m8NWvuAcJ9HAGeeeYZmpubRwVaUw4xigItRlGgxSgKtBhFgRajKNBiFAVajKJAi1EUaDGKAi1GUaDFKAq0GEWBFqMo0GKUCe+H9kQFsAaYA8SBLqAF+C/gi0AqGmHTsU2c7T87avsLc1+gprDG/4LG8PC+hznde3rU9i03b6G2pDaAikYK6hgGE+i/AGYBzcA5YAYwFygjFIEetLhkMcmC5NB4RmRGgNWM7daKW6kurB4aJ6KJAKsZze9j6H+gi3DD3A38+0XbI4RuArQqsYplZcsmvX/sgxhlPyvjwgMX6Lu5D0bdfp57a5NrWVm50vs3AqInopQ2ltK3oI+ur3dN6jXZHsPp8j/QvQP/ioDHgY+AT4BjQMr3asa16/wuDncfHhpvSG4Yd//o8Sixj2KUv1xOuipNx191eB7sHS07eL/9/aHxE/OeyPl7RE9EKd1WSvxQ3P1/lGbSgc72GE6X/4HOAP8J0a9F6Z/V756t/wS4AGwDTvpe0WXt79wPncPjzd/dPOFrHMvB7rWxP7OZ+dJM2h5vo+euHs9q3PPFnhHjXAfaPmNT9XQVDg7WwE9m4fuFzKqbNe7rIt+LQPnoY2heoAEOwStHXqFtbhub526GxUApcBfQGEhFY3oy+SQrzq0YGp9rODfu/oW7Cyn+TTEZOwMWdK7rpHdZr6c1NjQ1cNvG23Bu9mZJt0xlhrbH2yhrKsPqtrB7bVLXpWh/rH3816UzADxtP82Sq5aQKc94Ut+l/A+0DVwLseMxlh9b7k41uoC1QIHv1YwvCqkvT34eZLfaFP2uiM51nXT9WRdOiQHrBtrQc1cPPbf3UPjfhZQ1lZH6cmrC4+Icc6Af+mf1kynzJ8wQRKCjwKPw7JlnWXhqoTsn+6OB/9bsezU51bu8l5YlLe4HXNNEhoMdtg/vF/M/0P3AuxCribHn+j0QA9qB94B3fK8m90wM88VC/v0F86Hwl1BPPfOYx2pW+17CRH5c++OgS5jQ1iVbAYjVx+inP+BqRgvqGIb4l4dI9hRoMYoCLUZRoMUoCrQYRYEWo/ge6CqqeJEXmcc8AF7iJW7kRr/LyHv2z20i/+BeFI78KIL9z7Z7SfQK53ugEyRYxvDthItZzNVc7XcZec86ZGHtd28Wsk5Y2HsUaAgg0Mc4xj72kSYNQCut7GKX32XkvfQjafevrIBT6JB+NB3UrWahEsgc+jVeI0WKbrp5lVfJ6NSSvTngLHNwLAfi4NxjwI1QORBIoD/kQw5ykHbadXaehvQj7m85nZ2HBXYY6qknSlRn5+mYA/1b+6Ey6ELCI7BAdzG5R3hkAsmJd7mS6Dq0GGUKbd32eliOyGRZOI6TfVs3y7Iesyxrr2VZe+GMN7WJ5EjWZ+jGxr/3sJypG2xq6ctiGFPmHusDB/4n4Doub9GiWwa+CvNxdE3pDC2STxRoMYoCLUZRoMUoCrQYRYEWoyjQYhQFWoyiQItRFGgxigItRlGgxSgKtBhFgRaj+P4IVuibWn4PKAeagA8GttUAfw30AC8GUtUo63au41T3qVHbm1Y1MT8xP4CKLhHQcQzsmcJ8aGqZD+68+k6uLbl2aFxRUBFgNcELLNB+N2Q01fq561l9Tfi6IAQlsEDvOr+Lw52Hh2bxXvevy9oi3F+R4LZuDqntH29n79nh5zyfuumpAKsZg8/HMbBA7+/cP2IcukCHYBo6GbtbdkPL8Dh0gfb5OAYW6Od+9Ryr3lnF542fB1XC+Mb6MBNCDU0NfPX7XyW2NBZ0KWPz+Tjqsp0YRYEWoyjQYhTf59CDDRkr/zWkKwy+PMa2j4F6f8uYyFtr3gKgrb4t4EouI6DjqDO0GEWBFqMo0GIUBVqMokCLURRoMYrvgY5+FCX5N0lin7p/qk1uTFL4TqHfZeS9zh900nane8mu81udtK9vx0mpE5bvgc4kMli9w8v6WimLdGXa7zLynp203RvlwW24GUOdsAgi0DMzdN/VjRN1cHBIzU2RWpDyu4y8F38kDpGBQREUPVmEZYV/kXKvBTKHvnD/BXeB+ALoeKgjiBLynl1hE/9mHGywZ9tEb9PpGQIKdGZmhq4/7aLv+j6dnach/kgcq8Ki6Ps6Ow8K7Me6Y4POzNNlV9gk3k4EXUao6LKdGEWBFqOo8abkKTXelCtA1mfonTtf8LCcqVuz5h4gP5panjw5esWjsKiuvmbgq/BfNVHjTTGeAi1GUaDFKAq0GEWBFqMo0GIUBVqMokCLURRoMYoCLUZRoMUoCrQYRYEWoyjQYhTfnyl8eN/DnO49PWr7lpu3UFtS63c5Ywp9U0tg+bblfHrh01Hbd96/k69UfSWAii5xpTXevLXiVqoLq4fGiegED3s6wP8BtQyvR+GxfGhqefecu6mZUTM0riwK6ULyPgks0GuTa1lZuXLyL/gYYt+J4VQ6pDemcVY7ngc7H5pa1i2o496ae4MuIzQCC/SOUzs4+NuD0OuOv3342+Pub7VYOJaDdc4i8qMIvA7pJ9M4y7xbz2370e28d/o9rIj7YEToegAC2/Zv492T7w6Nn1/xfIDVjOFKaby5p30PDM842PTTTZN/cR+QAusPlqeB3t22Gy5qYRLGQP/67K/h7PA4dIG+Yhpvzn9uxJQjtXOCFZSOQfQ7UYiAs8SddjDH2xobmhpYVbmK0p+UevtG09DQ1EDdP9VRvKI46FLG5nPjzfxZEK0aMg9myKzOeB5kyV/5E+giyGzIBF2FhJz+sCJG8f0MvXXJVr/fMmsjmlpmcWXRT7//y98DcKT+SMCVXIYab4pMnwItRlGgxSgKtBhFgRajKNBjcHod0ifdVnNOm0PmVPiuf6fb06ROuH9dTZ1K0X+mP+CKwkGBHkPPGz10fN3tAZM+lKZ9XTv9H4QrMJ9t+IzmO5oB+PzvPqf5tmacPjXeVKDHEFsdG3GF3qqwiNT6dBP2JJV9owwrOrA8cgaKby/GKgj/ms5eU6DHEJ0fJboo6h6dIijcVIgVC1dYEnUJrLhbk1VgcdUzVwVcUTgo0JdR+N1CsMEqtCi4ryDockax4zZVT1WBDcV3FBOfHw+6pFDIn5uTfBZdEKXgmwVEF0dDd3YelKhL0LW7i6rNVUGXEhoK9DiKnw7pPcYD7LjN7NdnB11GqGjKIUZRoMUoarwpeUqNN+UKMIUz9D4Py5kO9/vIh6aWWRxy31lD57xwXtm5mBpvivEUaDGKAi1GUaDFKAq0GEWBFqMo0GIUBVqMokCLURRoMYoCLUZRoMUoCrQYxd9HsALqXZet0PcABGperuGT85+M2n7gWwe4ZdYtAVQ0htnA7cB1QBHQBbTg3lL/B2/eUs8UjiMfegDed8N91FYMNyy9qjgkyxksBB7AnQOcAY4CcdyQ34QCHYRsewCmTqXoeLODxIMJImX+LEyzcdFG1i9YP+n9z797nu5j3SQfTGJHPZpxxoD7cMN8ENgODK6mZgEePqQeTKB97l03VY0fNGbVA/DCLy9w5vkznP3hWWY+MZOKv63wPNivH3idXR/vGhq/vHaspfOHHX/hOK2/aKV5czNfeulL3gT7OmDwgfnfMRxmcJ/D8PDBp2ACHY522RN6+/jbI8YPPfDQxC+KgNPp0PqTVlpfaWX2T2dTcnuJRxXCm0ffHDFef+/kztZ9J/s4+vhRmn/QzNJ9Sym4OoeL6Vz87Q72ebwbdz49qD53b3exYALtc++6qXr1xle5qeEmnN6BZ6Ym+EHsO9Ln/ppNAxGIJqNEZnp7hn7jujdY+C8LhzfcOP7+XYe63C8s3DXxFniwJl7nRV/PAFqB48D7wB/n9q0upTn0OKKzo9TsqJn0/l/82xe0PNtCbF6M5D8mKVlTgmV5+2xe+Ypylj+6fNL7H/zGQc69eY7y1eXU/rCWssVluS/qBO4VjWLgDuA/cD8UtqNA55PEnyeIXx+naEWR50Geqhu23EBffR9lizwI8qAU8BZwP+7npWuAT4GEd285SIHOIbvYpnhluJcPi1fHiVf7sLDj/+KekVfifki8BXcq8iFwyLu31TIGPtIyBrmlZQzEeAq0GEWBFqMo0GIUBVqMokCLURRoMYoCLUZRoMUoCrQYRYEWoyjQYhQFWoyiQItRFGgxSpb3Q1sdwBHvypm2KuBs0EVMQDXmxlzHcUYtQpLtEytHHMdZmqOCcs6yrL1hrg9Uo9c05RCjKNBilGwD/ZonVeRO2OsD1eiprD4UioSdphxiFAVajKJAi1EUaDGKAi1G+X/shoM3Mha1dgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "after iteration 5\n",
            "iter    0   |   diff: 0.59049   |   V(start): 0.590 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAANvklEQVR4nO3df2wU553H8fczO+td/1wbm21iIDE1xG7aXEIMJIIkgJJQwkUpF9EKN4e4lDaFtiht2iRtTrqg+NRcq7aKmlYop8vlKk5gqWlRpYijNGmoaBTREojC8TPBofwMtjHG9tpe74/pH4NtjI3ttXd+7MP39Q9+VrPar4ePn52dnXm+yrIshNCF4XUBQmSTBFpoRQIttCKBFlqRQAutmGNtoJR6AnjCHhXWQa3DJQkxtqKio3R2dqqrH1eZnLZTaq61adNXs1pYtqxfvw6ArVsbPa7k2urrVwGwc+cfPK7k2pYufRDw934EeO6552hqahoWaDnkEFqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00MqY10M7oSXZwm+7fstHfR/Ra/VSZBRRaVayqngVU82pXpQ0xIbjG2hNtg57/MWbX6QqXOV+QSNY/d5qzsfPD3t80+2bqC6s9qCiobzah54E+pVLr3AmeYaavBqigSjtqXY+THzIpfQlpuJ9oPvdWXgn0bzowLgkUOJhNSO7q+wuKsOVA+OIGfGwmuHc3oeuBzqWjnEmeYZ8lc+TpU+ilH2NdsJKYOGvNUIWRxYzr3jeuLcPHglS/L/FdK3sou/2Phh2+Xn2LYsuY2H5QudfCDBPmRRtLaKvto/uR7rH9ZxM9+FkuR7osAoTUiF6rB5+2PZDavJqmBWcxWdCnyGkQm6XM6pdl3ZxqOfQwHhNdM2o25snTYIfByl9qZRURYrOf+50PNg7mnfwQccHA+P1M9dn/TXMUyZFW4oIHQxBAkgx7kBnug8ny/VAB1SA1SWr2XJxC6eTpzmdPM1bvEWJUcL60vVUBavcLuma9sX2QWxw/OyTz475HEtZGHED44zBlB9NoX1dO72Leh2rcc/FPUPG2Q600WJQ8UwFFhbq8l9m+IMwN9TfMOrzAt8OQOnwfahdoAHqwnU88t+PcDh0mLfXvc07Pe/Qke5ge9d2vlH2DS9KGtFT0adYcGHBwPhCw4VRtw/vDlPwxwLSRhoUxJbHiM+LO1pjQ2MDd6+9G+t2Zw7X0uVp2te1U9xYjOpRGHGDxIwEHU90jP68VBqAZ4xnqJtaR7o07Uh9V3M90CkrxceJj5mZmsn84/MpKSqhSBXxetfrxC1n//MzZkJiVmLcmxttBvl/yie2PEb3P3ZjFfrrM8GEGNC7qJfee3oJ/zlMcWMxiVmJMfeLddyCJCRvSJIudifM4EGgE1aCn178Kb/+0q+ZfWo2vR29vN/7PgC1odxeIiE+P05zXTMEvK7EAYHBYPv52wvXAx1UQe4vuJ8TrSfYM3sPvT29lAXKWBRexNKCpW6Xk306hvlKPv/9PPlQuLJ4JTN/M5PQuRBHNh1xu4QxvVz9stcljGlz3WYAghuDJEl6XM1wXu1DH795CJE5CbTQigRaaEUCLbQigRZakUALrbgeaPOiyfSXpxM6Z1+INP3l6YSPh90uI+cZrxsEfmCfFA78PIDxEwPc+0LOt1wPdKArQOHhwoFx4dFCgm1Bt8vIeeqgQu2zLxZSpxTGHgk0eBDo+Iw4sdoYlrKvc0gWJ+ms63S7jJyXejwFl+cBK2yR+krKo0vN/MWTY+iWf2rBMi3SeWmaH22WI/mJuAmseZY9MYTAelCDC6GywJMoxWfE6Z7VTaowJbPzJKQeT9n/yuw8wLPdcPZrZ1EpJbPzZNwEyc1JKPe6EP/wLNDpfPkEkxXRsTe5nsj8KLSScVs32OtgOUKMl8KyrMzbuimlnlBK7VVK7YUWZ2oTIksynqG3bv2eg+VMXH9TS1cWw5gwe1/v3/++x3Vc25w5d1z+yc/70TahGVqIXCKBFlqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFddvwfJ9U8tvA6VAI9C/dHUV8C9AL/AfnlQ1zPKdyznXc27Y442LG6mJ1HhQ0VU82o+e3VOYC00tc8F9n7qP6YXTB8ZleWUeVuM9zwLtdkNGXa24eQVLblzidRm+4Vmgd13axaHYoYGjeKf712VsDvZbJICP3zy2ndjG3tbB+zyfvu1pD6sZgcv70bNA74vtGzL2XaB9cBg6Hrubd0Pz4Nh3gXZ5P3oW6Of/8DyL31nMJ1s/8aqE0Y30YcaHGhob+Px3P09wrk8XvHR5P8ppO6EVCbTQigRaaMX1Y+j+hozl/+XTFQZfGuGxE8BGd8sYy/al2wFo39jucSXX4NF+lBlaaEUCLbQigRZakUALrUighVYk0EIr7jfe/Ngk+tUowdP2V7XRtVHC70jjzUzFvh+j/T77lF3s6zE6VnRgJaQTluuBTkfSqPjgsr4qoUiVp9wuI+cZUcO+UB7shptBpBMWXgR6SpqeRT1YpoWFReLmBInahNtl5LzQ4yEIXB7kQ/5T+Sjl/0XKnebJMXTXo132AvF50PmY9CmcCKPMIPTFEBhgTDMw75bpGTwKdHpKmu77u+mb3Sez8ySEHg+hyhT535XZuZ9nf9ada2RmniyjzCDyZsTrMnxFTtsJrUighVak8abIUdJ4U1wHMp6hd+580cFyJm7p0geB3Ghqefbs8BWP/KKy8sbLP/n/rIk03hTak0ALrUighVYk0EIrEmihFQm00IoEWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCK6/cUrn5vNefj54c9vun2TVQXVrtdzoh839QSmL9lPqe7Tg97fOejO/lcxec8qOgq11vjzbvK7qIyXDkwjphj3OxpAR8C1QyuR+GwXGhq+cBND1BVUjUwLs/36ULyLvEs0Muiy1hYvnD8TzgBwW8FscotUmtTWEssx4OdC00t62vreajqIa/L8A3PAr3j3A4OvH0A4vb4m4e+Oer2qllhKQt1QRH4eQBehdRTKax5zq3ntu3YNv56/q+ogH1jhO96AAJb9m3h3bPvDoxfWPCCh9WM4HppvLmnYw8MHnGw4Vcbxv/kPiAB6rByNNC723fDFS1M/Bjot1rfgtbBse8Cfd003qx5fsghR2LnGCsoHQfzWyYEwKqzDzu4ydkaGxobWFy+mKJfFDn7QpPQ0NhA/b/XU7CgwOtSRuZy483cWRCtEtKr0qSXpB0PsshduRPofEivSXtdhfA5+WJFaMX1GXpz3Wa3XzJjQ5paZnBm0U1/+fJfADi68ajHlVyDNN4UYvIk0EIrEmihFQm00IoEWmhFAj0CK26ROmu3mrPaLdLn/Hf+O9WRInHK/nY1cS5BsiXpcUX+IIEeQe9rvXQ+YveASR1M0bG8g+QRfwXmzJozNN3bBMAn3/mEprubsPqk8aYEegTBJcEhZ+hVmSJQ7dJF2ONU/IVilHl5eeQ0FNxTgMrz/5rOTpNAj8CsMTHnmPbeyYfwhjAq6K+wROojqJBdk8pTTH1uqscV+YME+hrCT4bBABVW5D2c53U5wxghg4qnK8CAgnsLCNWEvC7JF3Ln4iSXmbUmeV/Mw7zT9N3s3C9SH6F7dzcVz1Z4XYpvSKBHUfCMT68xvswIGUx7dZrXZfiKHHIIrUighVak8abIUdJ4U1wHJjBDv+dgOZNh/x650NQyg13uOjUw5/nzzM6VpPGm0J4EWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCKBFpoRQIttCKBFlqRQAutuHsLlke96zLl+x6AQNVLVfzt0t+GPb7/6/u544Y7PKhoBNOAe4AZQD7QDTRjX1J/2JmXlHsKR5ELPQAfvuVhqssGG5ZOLfDJcga3AiuxjwFagGNACDvktyGB9kKmPQAT5xJ0vtFJZFWEQLE7C9OsnbOWFbUrxr39pXcv0XO8h+iqKIbp0BFnEHgYO8wHgG1A/2pqCnDwJnVvAu1y77qJ2npka0Y9ALt+30XLCy20/riVKeunUPa1MseD/er+V9l1YtfA+KVlIy2dP+jkiydp+782mp5t4tM/+rQzwZ4B9N8w/ycGwwz2fRgO3vjkTaD90S57TG+efHPI+LGVj439pABYMYu2X7TR9ss2pv1qGoX3FDpUIbxx7I0h4xUPjW+27jvbx7F1x2j6fhNz35tL3qeyuJjOlb9uf5/HB7CPp/ttzN7LXcmbQLvcu26iXvnsK9zWcBtW/PI9U2P8IfYd7bPfZlNAAMyoSWCKszP0azNe49af3Tr4wGdH3777YLf9g8JeE6/WgTXxYlf8XAK0ASeBD4B/yO5LXU2OoUdhTjOp2lE17u0v/s9Fmv+1meDMINF/i1K4tBClnL03r3RBKfO/Mn/c2x/4wgEuvHGB0iWlVP+4muI7i7Nf1CnsMxoFwL3A77A/FHYggc4lkS9FCM0Okb8g3/EgT9Qtm26hb2MfxXMcCHK/BLAdeBT789KNwGkg4txL9pNAZ5FRYFCw0N/Lh4UqQ4QqXVjY8f+xZ+SF2B8S78A+FPkIOOjcy8oyBi6SZQyyS5YxENqTQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00IoEWmglw+uhVSdw1LlyJq0CaPW6iDFIjdlxs2VZwxYhyfSOlaOWZc3NUkFZp5Ta6+f6QGp0mhxyCK1IoIVWMg30fzpSRfb4vT6QGh2V0YdCIfxODjmEViTQQisSaKEVCbTQigRaaOXvDfHJlPquMXIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "after iteration 6\n",
            "iter    0   |   diff: 0.00000   |   V(start): 0.590 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAANvklEQVR4nO3df2wU553H8fczO+td/1wbm21iIDE1xG7aXEIMJIIkgJJQwkUpF9EKN4e4lDaFtiht2iRtTrqg+NRcq7aKmlYop8vlKk5gqWlRpYijNGmoaBTREojC8TPBofwMtjHG9tpe74/pH4NtjI3ttXd+7MP39Q9+VrPar4ePn52dnXm+yrIshNCF4XUBQmSTBFpoRQIttCKBFlqRQAutmGNtoJR6AnjCHhXWQa3DJQkxtqKio3R2dqqrH1eZnLZTaq61adNXs1pYtqxfvw6ArVsbPa7k2urrVwGwc+cfPK7k2pYufRDw934EeO6552hqahoWaDnkEFqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00MqY10M7oSXZwm+7fstHfR/Ra/VSZBRRaVayqngVU82pXpQ0xIbjG2hNtg57/MWbX6QqXOV+QSNY/d5qzsfPD3t80+2bqC6s9qCiobzah54E+pVLr3AmeYaavBqigSjtqXY+THzIpfQlpuJ9oPvdWXgn0bzowLgkUOJhNSO7q+wuKsOVA+OIGfGwmuHc3oeuBzqWjnEmeYZ8lc+TpU+ilH2NdsJKYOGvNUIWRxYzr3jeuLcPHglS/L/FdK3sou/2Phh2+Xn2LYsuY2H5QudfCDBPmRRtLaKvto/uR7rH9ZxM9+FkuR7osAoTUiF6rB5+2PZDavJqmBWcxWdCnyGkQm6XM6pdl3ZxqOfQwHhNdM2o25snTYIfByl9qZRURYrOf+50PNg7mnfwQccHA+P1M9dn/TXMUyZFW4oIHQxBAkgx7kBnug8ny/VAB1SA1SWr2XJxC6eTpzmdPM1bvEWJUcL60vVUBavcLuma9sX2QWxw/OyTz475HEtZGHED44zBlB9NoX1dO72Leh2rcc/FPUPG2Q600WJQ8UwFFhbq8l9m+IMwN9TfMOrzAt8OQOnwfahdoAHqwnU88t+PcDh0mLfXvc07Pe/Qke5ge9d2vlH2DS9KGtFT0adYcGHBwPhCw4VRtw/vDlPwxwLSRhoUxJbHiM+LO1pjQ2MDd6+9G+t2Zw7X0uVp2te1U9xYjOpRGHGDxIwEHU90jP68VBqAZ4xnqJtaR7o07Uh9V3M90CkrxceJj5mZmsn84/MpKSqhSBXxetfrxC1n//MzZkJiVmLcmxttBvl/yie2PEb3P3ZjFfrrM8GEGNC7qJfee3oJ/zlMcWMxiVmJMfeLddyCJCRvSJIudifM4EGgE1aCn178Kb/+0q+ZfWo2vR29vN/7PgC1odxeIiE+P05zXTMEvK7EAYHBYPv52wvXAx1UQe4vuJ8TrSfYM3sPvT29lAXKWBRexNKCpW6Xk306hvlKPv/9PPlQuLJ4JTN/M5PQuRBHNh1xu4QxvVz9stcljGlz3WYAghuDJEl6XM1wXu1DH795CJE5CbTQigRaaEUCLbQigRZakUALrbgeaPOiyfSXpxM6Z1+INP3l6YSPh90uI+cZrxsEfmCfFA78PIDxEwPc+0LOt1wPdKArQOHhwoFx4dFCgm1Bt8vIeeqgQu2zLxZSpxTGHgk0eBDo+Iw4sdoYlrKvc0gWJ+ms63S7jJyXejwFl+cBK2yR+krKo0vN/MWTY+iWf2rBMi3SeWmaH22WI/mJuAmseZY9MYTAelCDC6GywJMoxWfE6Z7VTaowJbPzJKQeT9n/yuw8wLPdcPZrZ1EpJbPzZNwEyc1JKPe6EP/wLNDpfPkEkxXRsTe5nsj8KLSScVs32OtgOUKMl8KyrMzbuimlnlBK7VVK7YUWZ2oTIksynqG3bv2eg+VMXH9TS1cWw5gwe1/v3/++x3Vc25w5d1z+yc/70TahGVqIXCKBFlqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFddvwfJ9U8tvA6VAI9C/dHUV8C9AL/AfnlQ1zPKdyznXc27Y442LG6mJ1HhQ0VU82o+e3VOYC00tc8F9n7qP6YXTB8ZleWUeVuM9zwLtdkNGXa24eQVLblzidRm+4Vmgd13axaHYoYGjeKf712VsDvZbJICP3zy2ndjG3tbB+zyfvu1pD6sZgcv70bNA74vtGzL2XaB9cBg6Hrubd0Pz4Nh3gXZ5P3oW6Of/8DyL31nMJ1s/8aqE0Y30YcaHGhob+Px3P09wrk8XvHR5P8ppO6EVCbTQigRaaMX1Y+j+hozl/+XTFQZfGuGxE8BGd8sYy/al2wFo39jucSXX4NF+lBlaaEUCLbQigRZakUALrUighVYk0EIr7jfe/Ngk+tUowdP2V7XRtVHC70jjzUzFvh+j/T77lF3s6zE6VnRgJaQTluuBTkfSqPjgsr4qoUiVp9wuI+cZUcO+UB7shptBpBMWXgR6SpqeRT1YpoWFReLmBInahNtl5LzQ4yEIXB7kQ/5T+Sjl/0XKnebJMXTXo132AvF50PmY9CmcCKPMIPTFEBhgTDMw75bpGTwKdHpKmu77u+mb3Sez8ySEHg+hyhT535XZuZ9nf9ada2RmniyjzCDyZsTrMnxFTtsJrUighVak8abIUdJ4U1wHMp6hd+580cFyJm7p0geB3Ghqefbs8BWP/KKy8sbLP/n/rIk03hTak0ALrUighVYk0EIrEmihFQm00IoEWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCK6/cUrn5vNefj54c9vun2TVQXVrtdzoh839QSmL9lPqe7Tg97fOejO/lcxec8qOgq11vjzbvK7qIyXDkwjphj3OxpAR8C1QyuR+GwXGhq+cBND1BVUjUwLs/36ULyLvEs0Muiy1hYvnD8TzgBwW8FscotUmtTWEssx4OdC00t62vreajqIa/L8A3PAr3j3A4OvH0A4vb4m4e+Oer2qllhKQt1QRH4eQBehdRTKax5zq3ntu3YNv56/q+ogH1jhO96AAJb9m3h3bPvDoxfWPCCh9WM4HppvLmnYw8MHnGw4Vcbxv/kPiAB6rByNNC723fDFS1M/Bjot1rfgtbBse8Cfd003qx5fsghR2LnGCsoHQfzWyYEwKqzDzu4ydkaGxobWFy+mKJfFDn7QpPQ0NhA/b/XU7CgwOtSRuZy483cWRCtEtKr0qSXpB0PsshduRPofEivSXtdhfA5+WJFaMX1GXpz3Wa3XzJjQ5paZnBm0U1/+fJfADi68ajHlVyDNN4UYvIk0EIrEmihFQm00IoEWmhFAj0CK26ROmu3mrPaLdLn/Hf+O9WRInHK/nY1cS5BsiXpcUX+IIEeQe9rvXQ+YveASR1M0bG8g+QRfwXmzJozNN3bBMAn3/mEprubsPqk8aYEegTBJcEhZ+hVmSJQ7dJF2ONU/IVilHl5eeQ0FNxTgMrz/5rOTpNAj8CsMTHnmPbeyYfwhjAq6K+wROojqJBdk8pTTH1uqscV+YME+hrCT4bBABVW5D2c53U5wxghg4qnK8CAgnsLCNWEvC7JF3Ln4iSXmbUmeV/Mw7zT9N3s3C9SH6F7dzcVz1Z4XYpvSKBHUfCMT68xvswIGUx7dZrXZfiKHHIIrUighVak8abIUdJ4U1wHJjBDv+dgOZNh/x650NQyg13uOjUw5/nzzM6VpPGm0J4EWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCKBFpoRQIttCKBFlqRQAutuHsLlke96zLl+x6AQNVLVfzt0t+GPb7/6/u544Y7PKhoBNOAe4AZQD7QDTRjX1J/2JmXlHsKR5ELPQAfvuVhqssGG5ZOLfDJcga3AiuxjwFagGNACDvktyGB9kKmPQAT5xJ0vtFJZFWEQLE7C9OsnbOWFbUrxr39pXcv0XO8h+iqKIbp0BFnEHgYO8wHgG1A/2pqCnDwJnVvAu1y77qJ2npka0Y9ALt+30XLCy20/riVKeunUPa1MseD/er+V9l1YtfA+KVlIy2dP+jkiydp+782mp5t4tM/+rQzwZ4B9N8w/ycGwwz2fRgO3vjkTaD90S57TG+efHPI+LGVj439pABYMYu2X7TR9ss2pv1qGoX3FDpUIbxx7I0h4xUPjW+27jvbx7F1x2j6fhNz35tL3qeyuJjOlb9uf5/HB7CPp/ttzN7LXcmbQLvcu26iXvnsK9zWcBtW/PI9U2P8IfYd7bPfZlNAAMyoSWCKszP0azNe49af3Tr4wGdH3777YLf9g8JeE6/WgTXxYlf8XAK0ASeBD4B/yO5LXU2OoUdhTjOp2lE17u0v/s9Fmv+1meDMINF/i1K4tBClnL03r3RBKfO/Mn/c2x/4wgEuvHGB0iWlVP+4muI7i7Nf1CnsMxoFwL3A77A/FHYggc4lkS9FCM0Okb8g3/EgT9Qtm26hb2MfxXMcCHK/BLAdeBT789KNwGkg4txL9pNAZ5FRYFCw0N/Lh4UqQ4QqXVjY8f+xZ+SF2B8S78A+FPkIOOjcy8oyBi6SZQyyS5YxENqTQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00IoEWmglw+uhVSdw1LlyJq0CaPW6iDFIjdlxs2VZwxYhyfSOlaOWZc3NUkFZp5Ta6+f6QGp0mhxyCK1IoIVWMg30fzpSRfb4vT6QGh2V0YdCIfxODjmEViTQQisSaKEVCbTQigRaaOXvDfHJlPquMXIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "after iteration 7\n",
            "iter    0   |   diff: 0.00000   |   V(start): 0.590 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAANvklEQVR4nO3df2wU553H8fczO+td/1wbm21iIDE1xG7aXEIMJIIkgJJQwkUpF9EKN4e4lDaFtiht2iRtTrqg+NRcq7aKmlYop8vlKk5gqWlRpYijNGmoaBTREojC8TPBofwMtjHG9tpe74/pH4NtjI3ttXd+7MP39Q9+VrPar4ePn52dnXm+yrIshNCF4XUBQmSTBFpoRQIttCKBFlqRQAutmGNtoJR6AnjCHhXWQa3DJQkxtqKio3R2dqqrH1eZnLZTaq61adNXs1pYtqxfvw6ArVsbPa7k2urrVwGwc+cfPK7k2pYufRDw934EeO6552hqahoWaDnkEFqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00MqY10M7oSXZwm+7fstHfR/Ra/VSZBRRaVayqngVU82pXpQ0xIbjG2hNtg57/MWbX6QqXOV+QSNY/d5qzsfPD3t80+2bqC6s9qCiobzah54E+pVLr3AmeYaavBqigSjtqXY+THzIpfQlpuJ9oPvdWXgn0bzowLgkUOJhNSO7q+wuKsOVA+OIGfGwmuHc3oeuBzqWjnEmeYZ8lc+TpU+ilH2NdsJKYOGvNUIWRxYzr3jeuLcPHglS/L/FdK3sou/2Phh2+Xn2LYsuY2H5QudfCDBPmRRtLaKvto/uR7rH9ZxM9+FkuR7osAoTUiF6rB5+2PZDavJqmBWcxWdCnyGkQm6XM6pdl3ZxqOfQwHhNdM2o25snTYIfByl9qZRURYrOf+50PNg7mnfwQccHA+P1M9dn/TXMUyZFW4oIHQxBAkgx7kBnug8ny/VAB1SA1SWr2XJxC6eTpzmdPM1bvEWJUcL60vVUBavcLuma9sX2QWxw/OyTz475HEtZGHED44zBlB9NoX1dO72Leh2rcc/FPUPG2Q600WJQ8UwFFhbq8l9m+IMwN9TfMOrzAt8OQOnwfahdoAHqwnU88t+PcDh0mLfXvc07Pe/Qke5ge9d2vlH2DS9KGtFT0adYcGHBwPhCw4VRtw/vDlPwxwLSRhoUxJbHiM+LO1pjQ2MDd6+9G+t2Zw7X0uVp2te1U9xYjOpRGHGDxIwEHU90jP68VBqAZ4xnqJtaR7o07Uh9V3M90CkrxceJj5mZmsn84/MpKSqhSBXxetfrxC1n//MzZkJiVmLcmxttBvl/yie2PEb3P3ZjFfrrM8GEGNC7qJfee3oJ/zlMcWMxiVmJMfeLddyCJCRvSJIudifM4EGgE1aCn178Kb/+0q+ZfWo2vR29vN/7PgC1odxeIiE+P05zXTMEvK7EAYHBYPv52wvXAx1UQe4vuJ8TrSfYM3sPvT29lAXKWBRexNKCpW6Xk306hvlKPv/9PPlQuLJ4JTN/M5PQuRBHNh1xu4QxvVz9stcljGlz3WYAghuDJEl6XM1wXu1DH795CJE5CbTQigRaaEUCLbQigRZakUALrbgeaPOiyfSXpxM6Z1+INP3l6YSPh90uI+cZrxsEfmCfFA78PIDxEwPc+0LOt1wPdKArQOHhwoFx4dFCgm1Bt8vIeeqgQu2zLxZSpxTGHgk0eBDo+Iw4sdoYlrKvc0gWJ+ms63S7jJyXejwFl+cBK2yR+krKo0vN/MWTY+iWf2rBMi3SeWmaH22WI/mJuAmseZY9MYTAelCDC6GywJMoxWfE6Z7VTaowJbPzJKQeT9n/yuw8wLPdcPZrZ1EpJbPzZNwEyc1JKPe6EP/wLNDpfPkEkxXRsTe5nsj8KLSScVs32OtgOUKMl8KyrMzbuimlnlBK7VVK7YUWZ2oTIksynqG3bv2eg+VMXH9TS1cWw5gwe1/v3/++x3Vc25w5d1z+yc/70TahGVqIXCKBFlqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFddvwfJ9U8tvA6VAI9C/dHUV8C9AL/AfnlQ1zPKdyznXc27Y442LG6mJ1HhQ0VU82o+e3VOYC00tc8F9n7qP6YXTB8ZleWUeVuM9zwLtdkNGXa24eQVLblzidRm+4Vmgd13axaHYoYGjeKf712VsDvZbJICP3zy2ndjG3tbB+zyfvu1pD6sZgcv70bNA74vtGzL2XaB9cBg6Hrubd0Pz4Nh3gXZ5P3oW6Of/8DyL31nMJ1s/8aqE0Y30YcaHGhob+Px3P09wrk8XvHR5P8ppO6EVCbTQigRaaMX1Y+j+hozl/+XTFQZfGuGxE8BGd8sYy/al2wFo39jucSXX4NF+lBlaaEUCLbQigRZakUALrUighVYk0EIr7jfe/Ngk+tUowdP2V7XRtVHC70jjzUzFvh+j/T77lF3s6zE6VnRgJaQTluuBTkfSqPjgsr4qoUiVp9wuI+cZUcO+UB7shptBpBMWXgR6SpqeRT1YpoWFReLmBInahNtl5LzQ4yEIXB7kQ/5T+Sjl/0XKnebJMXTXo132AvF50PmY9CmcCKPMIPTFEBhgTDMw75bpGTwKdHpKmu77u+mb3Sez8ySEHg+hyhT535XZuZ9nf9ada2RmniyjzCDyZsTrMnxFTtsJrUighVak8abIUdJ4U1wHMp6hd+580cFyJm7p0geB3Ghqefbs8BWP/KKy8sbLP/n/rIk03hTak0ALrUighVYk0EIrEmihFQm00IoEWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCK6/cUrn5vNefj54c9vun2TVQXVrtdzoh839QSmL9lPqe7Tg97fOejO/lcxec8qOgq11vjzbvK7qIyXDkwjphj3OxpAR8C1QyuR+GwXGhq+cBND1BVUjUwLs/36ULyLvEs0Muiy1hYvnD8TzgBwW8FscotUmtTWEssx4OdC00t62vreajqIa/L8A3PAr3j3A4OvH0A4vb4m4e+Oer2qllhKQt1QRH4eQBehdRTKax5zq3ntu3YNv56/q+ogH1jhO96AAJb9m3h3bPvDoxfWPCCh9WM4HppvLmnYw8MHnGw4Vcbxv/kPiAB6rByNNC723fDFS1M/Bjot1rfgtbBse8Cfd003qx5fsghR2LnGCsoHQfzWyYEwKqzDzu4ydkaGxobWFy+mKJfFDn7QpPQ0NhA/b/XU7CgwOtSRuZy483cWRCtEtKr0qSXpB0PsshduRPofEivSXtdhfA5+WJFaMX1GXpz3Wa3XzJjQ5paZnBm0U1/+fJfADi68ajHlVyDNN4UYvIk0EIrEmihFQm00IoEWmhFAj0CK26ROmu3mrPaLdLn/Hf+O9WRInHK/nY1cS5BsiXpcUX+IIEeQe9rvXQ+YveASR1M0bG8g+QRfwXmzJozNN3bBMAn3/mEprubsPqk8aYEegTBJcEhZ+hVmSJQ7dJF2ONU/IVilHl5eeQ0FNxTgMrz/5rOTpNAj8CsMTHnmPbeyYfwhjAq6K+wROojqJBdk8pTTH1uqscV+YME+hrCT4bBABVW5D2c53U5wxghg4qnK8CAgnsLCNWEvC7JF3Ln4iSXmbUmeV/Mw7zT9N3s3C9SH6F7dzcVz1Z4XYpvSKBHUfCMT68xvswIGUx7dZrXZfiKHHIIrUighVak8abIUdJ4U1wHJjBDv+dgOZNh/x650NQyg13uOjUw5/nzzM6VpPGm0J4EWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCKBFpoRQIttCKBFlqRQAutuHsLlke96zLl+x6AQNVLVfzt0t+GPb7/6/u544Y7PKhoBNOAe4AZQD7QDTRjX1J/2JmXlHsKR5ELPQAfvuVhqssGG5ZOLfDJcga3AiuxjwFagGNACDvktyGB9kKmPQAT5xJ0vtFJZFWEQLE7C9OsnbOWFbUrxr39pXcv0XO8h+iqKIbp0BFnEHgYO8wHgG1A/2pqCnDwJnVvAu1y77qJ2npka0Y9ALt+30XLCy20/riVKeunUPa1MseD/er+V9l1YtfA+KVlIy2dP+jkiydp+782mp5t4tM/+rQzwZ4B9N8w/ycGwwz2fRgO3vjkTaD90S57TG+efHPI+LGVj439pABYMYu2X7TR9ss2pv1qGoX3FDpUIbxx7I0h4xUPjW+27jvbx7F1x2j6fhNz35tL3qeyuJjOlb9uf5/HB7CPp/ttzN7LXcmbQLvcu26iXvnsK9zWcBtW/PI9U2P8IfYd7bPfZlNAAMyoSWCKszP0azNe49af3Tr4wGdH3777YLf9g8JeE6/WgTXxYlf8XAK0ASeBD4B/yO5LXU2OoUdhTjOp2lE17u0v/s9Fmv+1meDMINF/i1K4tBClnL03r3RBKfO/Mn/c2x/4wgEuvHGB0iWlVP+4muI7i7Nf1CnsMxoFwL3A77A/FHYggc4lkS9FCM0Okb8g3/EgT9Qtm26hb2MfxXMcCHK/BLAdeBT789KNwGkg4txL9pNAZ5FRYFCw0N/Lh4UqQ4QqXVjY8f+xZ+SF2B8S78A+FPkIOOjcy8oyBi6SZQyyS5YxENqTQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00IoEWmglw+uhVSdw1LlyJq0CaPW6iDFIjdlxs2VZwxYhyfSOlaOWZc3NUkFZp5Ta6+f6QGp0mhxyCK1IoIVWMg30fzpSRfb4vT6QGh2V0YdCIfxODjmEViTQQisSaKEVCbTQigRaaOXvDfHJlPquMXIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "after iteration 8\n",
            "iter    0   |   diff: 0.00000   |   V(start): 0.590 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAANvklEQVR4nO3df2wU553H8fczO+td/1wbm21iIDE1xG7aXEIMJIIkgJJQwkUpF9EKN4e4lDaFtiht2iRtTrqg+NRcq7aKmlYop8vlKk5gqWlRpYijNGmoaBTREojC8TPBofwMtjHG9tpe74/pH4NtjI3ttXd+7MP39Q9+VrPar4ePn52dnXm+yrIshNCF4XUBQmSTBFpoRQIttCKBFlqRQAutmGNtoJR6AnjCHhXWQa3DJQkxtqKio3R2dqqrH1eZnLZTaq61adNXs1pYtqxfvw6ArVsbPa7k2urrVwGwc+cfPK7k2pYufRDw934EeO6552hqahoWaDnkEFqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00MqY10M7oSXZwm+7fstHfR/Ra/VSZBRRaVayqngVU82pXpQ0xIbjG2hNtg57/MWbX6QqXOV+QSNY/d5qzsfPD3t80+2bqC6s9qCiobzah54E+pVLr3AmeYaavBqigSjtqXY+THzIpfQlpuJ9oPvdWXgn0bzowLgkUOJhNSO7q+wuKsOVA+OIGfGwmuHc3oeuBzqWjnEmeYZ8lc+TpU+ilH2NdsJKYOGvNUIWRxYzr3jeuLcPHglS/L/FdK3sou/2Phh2+Xn2LYsuY2H5QudfCDBPmRRtLaKvto/uR7rH9ZxM9+FkuR7osAoTUiF6rB5+2PZDavJqmBWcxWdCnyGkQm6XM6pdl3ZxqOfQwHhNdM2o25snTYIfByl9qZRURYrOf+50PNg7mnfwQccHA+P1M9dn/TXMUyZFW4oIHQxBAkgx7kBnug8ny/VAB1SA1SWr2XJxC6eTpzmdPM1bvEWJUcL60vVUBavcLuma9sX2QWxw/OyTz475HEtZGHED44zBlB9NoX1dO72Leh2rcc/FPUPG2Q600WJQ8UwFFhbq8l9m+IMwN9TfMOrzAt8OQOnwfahdoAHqwnU88t+PcDh0mLfXvc07Pe/Qke5ge9d2vlH2DS9KGtFT0adYcGHBwPhCw4VRtw/vDlPwxwLSRhoUxJbHiM+LO1pjQ2MDd6+9G+t2Zw7X0uVp2te1U9xYjOpRGHGDxIwEHU90jP68VBqAZ4xnqJtaR7o07Uh9V3M90CkrxceJj5mZmsn84/MpKSqhSBXxetfrxC1n//MzZkJiVmLcmxttBvl/yie2PEb3P3ZjFfrrM8GEGNC7qJfee3oJ/zlMcWMxiVmJMfeLddyCJCRvSJIudifM4EGgE1aCn178Kb/+0q+ZfWo2vR29vN/7PgC1odxeIiE+P05zXTMEvK7EAYHBYPv52wvXAx1UQe4vuJ8TrSfYM3sPvT29lAXKWBRexNKCpW6Xk306hvlKPv/9PPlQuLJ4JTN/M5PQuRBHNh1xu4QxvVz9stcljGlz3WYAghuDJEl6XM1wXu1DH795CJE5CbTQigRaaEUCLbQigRZakUALrbgeaPOiyfSXpxM6Z1+INP3l6YSPh90uI+cZrxsEfmCfFA78PIDxEwPc+0LOt1wPdKArQOHhwoFx4dFCgm1Bt8vIeeqgQu2zLxZSpxTGHgk0eBDo+Iw4sdoYlrKvc0gWJ+ms63S7jJyXejwFl+cBK2yR+krKo0vN/MWTY+iWf2rBMi3SeWmaH22WI/mJuAmseZY9MYTAelCDC6GywJMoxWfE6Z7VTaowJbPzJKQeT9n/yuw8wLPdcPZrZ1EpJbPzZNwEyc1JKPe6EP/wLNDpfPkEkxXRsTe5nsj8KLSScVs32OtgOUKMl8KyrMzbuimlnlBK7VVK7YUWZ2oTIksynqG3bv2eg+VMXH9TS1cWw5gwe1/v3/++x3Vc25w5d1z+yc/70TahGVqIXCKBFlqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFddvwfJ9U8tvA6VAI9C/dHUV8C9AL/AfnlQ1zPKdyznXc27Y442LG6mJ1HhQ0VU82o+e3VOYC00tc8F9n7qP6YXTB8ZleWUeVuM9zwLtdkNGXa24eQVLblzidRm+4Vmgd13axaHYoYGjeKf712VsDvZbJICP3zy2ndjG3tbB+zyfvu1pD6sZgcv70bNA74vtGzL2XaB9cBg6Hrubd0Pz4Nh3gXZ5P3oW6Of/8DyL31nMJ1s/8aqE0Y30YcaHGhob+Px3P09wrk8XvHR5P8ppO6EVCbTQigRaaMX1Y+j+hozl/+XTFQZfGuGxE8BGd8sYy/al2wFo39jucSXX4NF+lBlaaEUCLbQigRZakUALrUighVYk0EIr7jfe/Ngk+tUowdP2V7XRtVHC70jjzUzFvh+j/T77lF3s6zE6VnRgJaQTluuBTkfSqPjgsr4qoUiVp9wuI+cZUcO+UB7shptBpBMWXgR6SpqeRT1YpoWFReLmBInahNtl5LzQ4yEIXB7kQ/5T+Sjl/0XKnebJMXTXo132AvF50PmY9CmcCKPMIPTFEBhgTDMw75bpGTwKdHpKmu77u+mb3Sez8ySEHg+hyhT535XZuZ9nf9ada2RmniyjzCDyZsTrMnxFTtsJrUighVak8abIUdJ4U1wHMp6hd+580cFyJm7p0geB3Ghqefbs8BWP/KKy8sbLP/n/rIk03hTak0ALrUighVYk0EIrEmihFQm00IoEWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCK6/cUrn5vNefj54c9vun2TVQXVrtdzoh839QSmL9lPqe7Tg97fOejO/lcxec8qOgq11vjzbvK7qIyXDkwjphj3OxpAR8C1QyuR+GwXGhq+cBND1BVUjUwLs/36ULyLvEs0Muiy1hYvnD8TzgBwW8FscotUmtTWEssx4OdC00t62vreajqIa/L8A3PAr3j3A4OvH0A4vb4m4e+Oer2qllhKQt1QRH4eQBehdRTKax5zq3ntu3YNv56/q+ogH1jhO96AAJb9m3h3bPvDoxfWPCCh9WM4HppvLmnYw8MHnGw4Vcbxv/kPiAB6rByNNC723fDFS1M/Bjot1rfgtbBse8Cfd003qx5fsghR2LnGCsoHQfzWyYEwKqzDzu4ydkaGxobWFy+mKJfFDn7QpPQ0NhA/b/XU7CgwOtSRuZy483cWRCtEtKr0qSXpB0PsshduRPofEivSXtdhfA5+WJFaMX1GXpz3Wa3XzJjQ5paZnBm0U1/+fJfADi68ajHlVyDNN4UYvIk0EIrEmihFQm00IoEWmhFAj0CK26ROmu3mrPaLdLn/Hf+O9WRInHK/nY1cS5BsiXpcUX+IIEeQe9rvXQ+YveASR1M0bG8g+QRfwXmzJozNN3bBMAn3/mEprubsPqk8aYEegTBJcEhZ+hVmSJQ7dJF2ONU/IVilHl5eeQ0FNxTgMrz/5rOTpNAj8CsMTHnmPbeyYfwhjAq6K+wROojqJBdk8pTTH1uqscV+YME+hrCT4bBABVW5D2c53U5wxghg4qnK8CAgnsLCNWEvC7JF3Ln4iSXmbUmeV/Mw7zT9N3s3C9SH6F7dzcVz1Z4XYpvSKBHUfCMT68xvswIGUx7dZrXZfiKHHIIrUighVak8abIUdJ4U1wHJjBDv+dgOZNh/x650NQyg13uOjUw5/nzzM6VpPGm0J4EWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCKBFpoRQIttCKBFlqRQAutuHsLlke96zLl+x6AQNVLVfzt0t+GPb7/6/u544Y7PKhoBNOAe4AZQD7QDTRjX1J/2JmXlHsKR5ELPQAfvuVhqssGG5ZOLfDJcga3AiuxjwFagGNACDvktyGB9kKmPQAT5xJ0vtFJZFWEQLE7C9OsnbOWFbUrxr39pXcv0XO8h+iqKIbp0BFnEHgYO8wHgG1A/2pqCnDwJnVvAu1y77qJ2npka0Y9ALt+30XLCy20/riVKeunUPa1MseD/er+V9l1YtfA+KVlIy2dP+jkiydp+782mp5t4tM/+rQzwZ4B9N8w/ycGwwz2fRgO3vjkTaD90S57TG+efHPI+LGVj439pABYMYu2X7TR9ss2pv1qGoX3FDpUIbxx7I0h4xUPjW+27jvbx7F1x2j6fhNz35tL3qeyuJjOlb9uf5/HB7CPp/ttzN7LXcmbQLvcu26iXvnsK9zWcBtW/PI9U2P8IfYd7bPfZlNAAMyoSWCKszP0azNe49af3Tr4wGdH3777YLf9g8JeE6/WgTXxYlf8XAK0ASeBD4B/yO5LXU2OoUdhTjOp2lE17u0v/s9Fmv+1meDMINF/i1K4tBClnL03r3RBKfO/Mn/c2x/4wgEuvHGB0iWlVP+4muI7i7Nf1CnsMxoFwL3A77A/FHYggc4lkS9FCM0Okb8g3/EgT9Qtm26hb2MfxXMcCHK/BLAdeBT789KNwGkg4txL9pNAZ5FRYFCw0N/Lh4UqQ4QqXVjY8f+xZ+SF2B8S78A+FPkIOOjcy8oyBi6SZQyyS5YxENqTQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00IoEWmglw+uhVSdw1LlyJq0CaPW6iDFIjdlxs2VZwxYhyfSOlaOWZc3NUkFZp5Ta6+f6QGp0mhxyCK1IoIVWMg30fzpSRfb4vT6QGh2V0YdCIfxODjmEViTQQisSaKEVCbTQigRaaOXvDfHJlPquMXIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "after iteration 9\n",
            "iter    0   |   diff: 0.00000   |   V(start): 0.590 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAANvklEQVR4nO3df2wU553H8fczO+td/1wbm21iIDE1xG7aXEIMJIIkgJJQwkUpF9EKN4e4lDaFtiht2iRtTrqg+NRcq7aKmlYop8vlKk5gqWlRpYijNGmoaBTREojC8TPBofwMtjHG9tpe74/pH4NtjI3ttXd+7MP39Q9+VrPar4ePn52dnXm+yrIshNCF4XUBQmSTBFpoRQIttCKBFlqRQAutmGNtoJR6AnjCHhXWQa3DJQkxtqKio3R2dqqrH1eZnLZTaq61adNXs1pYtqxfvw6ArVsbPa7k2urrVwGwc+cfPK7k2pYufRDw934EeO6552hqahoWaDnkEFqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00MqY10M7oSXZwm+7fstHfR/Ra/VSZBRRaVayqngVU82pXpQ0xIbjG2hNtg57/MWbX6QqXOV+QSNY/d5qzsfPD3t80+2bqC6s9qCiobzah54E+pVLr3AmeYaavBqigSjtqXY+THzIpfQlpuJ9oPvdWXgn0bzowLgkUOJhNSO7q+wuKsOVA+OIGfGwmuHc3oeuBzqWjnEmeYZ8lc+TpU+ilH2NdsJKYOGvNUIWRxYzr3jeuLcPHglS/L/FdK3sou/2Phh2+Xn2LYsuY2H5QudfCDBPmRRtLaKvto/uR7rH9ZxM9+FkuR7osAoTUiF6rB5+2PZDavJqmBWcxWdCnyGkQm6XM6pdl3ZxqOfQwHhNdM2o25snTYIfByl9qZRURYrOf+50PNg7mnfwQccHA+P1M9dn/TXMUyZFW4oIHQxBAkgx7kBnug8ny/VAB1SA1SWr2XJxC6eTpzmdPM1bvEWJUcL60vVUBavcLuma9sX2QWxw/OyTz475HEtZGHED44zBlB9NoX1dO72Leh2rcc/FPUPG2Q600WJQ8UwFFhbq8l9m+IMwN9TfMOrzAt8OQOnwfahdoAHqwnU88t+PcDh0mLfXvc07Pe/Qke5ge9d2vlH2DS9KGtFT0adYcGHBwPhCw4VRtw/vDlPwxwLSRhoUxJbHiM+LO1pjQ2MDd6+9G+t2Zw7X0uVp2te1U9xYjOpRGHGDxIwEHU90jP68VBqAZ4xnqJtaR7o07Uh9V3M90CkrxceJj5mZmsn84/MpKSqhSBXxetfrxC1n//MzZkJiVmLcmxttBvl/yie2PEb3P3ZjFfrrM8GEGNC7qJfee3oJ/zlMcWMxiVmJMfeLddyCJCRvSJIudifM4EGgE1aCn178Kb/+0q+ZfWo2vR29vN/7PgC1odxeIiE+P05zXTMEvK7EAYHBYPv52wvXAx1UQe4vuJ8TrSfYM3sPvT29lAXKWBRexNKCpW6Xk306hvlKPv/9PPlQuLJ4JTN/M5PQuRBHNh1xu4QxvVz9stcljGlz3WYAghuDJEl6XM1wXu1DH795CJE5CbTQigRaaEUCLbQigRZakUALrbgeaPOiyfSXpxM6Z1+INP3l6YSPh90uI+cZrxsEfmCfFA78PIDxEwPc+0LOt1wPdKArQOHhwoFx4dFCgm1Bt8vIeeqgQu2zLxZSpxTGHgk0eBDo+Iw4sdoYlrKvc0gWJ+ms63S7jJyXejwFl+cBK2yR+krKo0vN/MWTY+iWf2rBMi3SeWmaH22WI/mJuAmseZY9MYTAelCDC6GywJMoxWfE6Z7VTaowJbPzJKQeT9n/yuw8wLPdcPZrZ1EpJbPzZNwEyc1JKPe6EP/wLNDpfPkEkxXRsTe5nsj8KLSScVs32OtgOUKMl8KyrMzbuimlnlBK7VVK7YUWZ2oTIksynqG3bv2eg+VMXH9TS1cWw5gwe1/v3/++x3Vc25w5d1z+yc/70TahGVqIXCKBFlqRQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFddvwfJ9U8tvA6VAI9C/dHUV8C9AL/AfnlQ1zPKdyznXc27Y442LG6mJ1HhQ0VU82o+e3VOYC00tc8F9n7qP6YXTB8ZleWUeVuM9zwLtdkNGXa24eQVLblzidRm+4Vmgd13axaHYoYGjeKf712VsDvZbJICP3zy2ndjG3tbB+zyfvu1pD6sZgcv70bNA74vtGzL2XaB9cBg6Hrubd0Pz4Nh3gXZ5P3oW6Of/8DyL31nMJ1s/8aqE0Y30YcaHGhob+Px3P09wrk8XvHR5P8ppO6EVCbTQigRaaMX1Y+j+hozl/+XTFQZfGuGxE8BGd8sYy/al2wFo39jucSXX4NF+lBlaaEUCLbQigRZakUALrUighVYk0EIr7jfe/Ngk+tUowdP2V7XRtVHC70jjzUzFvh+j/T77lF3s6zE6VnRgJaQTluuBTkfSqPjgsr4qoUiVp9wuI+cZUcO+UB7shptBpBMWXgR6SpqeRT1YpoWFReLmBInahNtl5LzQ4yEIXB7kQ/5T+Sjl/0XKnebJMXTXo132AvF50PmY9CmcCKPMIPTFEBhgTDMw75bpGTwKdHpKmu77u+mb3Sez8ySEHg+hyhT535XZuZ9nf9ada2RmniyjzCDyZsTrMnxFTtsJrUighVak8abIUdJ4U1wHMp6hd+580cFyJm7p0geB3Ghqefbs8BWP/KKy8sbLP/n/rIk03hTak0ALrUighVYk0EIrEmihFQm00IoEWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCK6/cUrn5vNefj54c9vun2TVQXVrtdzoh839QSmL9lPqe7Tg97fOejO/lcxec8qOgq11vjzbvK7qIyXDkwjphj3OxpAR8C1QyuR+GwXGhq+cBND1BVUjUwLs/36ULyLvEs0Muiy1hYvnD8TzgBwW8FscotUmtTWEssx4OdC00t62vreajqIa/L8A3PAr3j3A4OvH0A4vb4m4e+Oer2qllhKQt1QRH4eQBehdRTKax5zq3ntu3YNv56/q+ogH1jhO96AAJb9m3h3bPvDoxfWPCCh9WM4HppvLmnYw8MHnGw4Vcbxv/kPiAB6rByNNC723fDFS1M/Bjot1rfgtbBse8Cfd003qx5fsghR2LnGCsoHQfzWyYEwKqzDzu4ydkaGxobWFy+mKJfFDn7QpPQ0NhA/b/XU7CgwOtSRuZy483cWRCtEtKr0qSXpB0PsshduRPofEivSXtdhfA5+WJFaMX1GXpz3Wa3XzJjQ5paZnBm0U1/+fJfADi68ajHlVyDNN4UYvIk0EIrEmihFQm00IoEWmhFAj0CK26ROmu3mrPaLdLn/Hf+O9WRInHK/nY1cS5BsiXpcUX+IIEeQe9rvXQ+YveASR1M0bG8g+QRfwXmzJozNN3bBMAn3/mEprubsPqk8aYEegTBJcEhZ+hVmSJQ7dJF2ONU/IVilHl5eeQ0FNxTgMrz/5rOTpNAj8CsMTHnmPbeyYfwhjAq6K+wROojqJBdk8pTTH1uqscV+YME+hrCT4bBABVW5D2c53U5wxghg4qnK8CAgnsLCNWEvC7JF3Ln4iSXmbUmeV/Mw7zT9N3s3C9SH6F7dzcVz1Z4XYpvSKBHUfCMT68xvswIGUx7dZrXZfiKHHIIrUighVak8abIUdJ4U1wHJjBDv+dgOZNh/x650NQyg13uOjUw5/nzzM6VpPGm0J4EWmhFAi20IoEWWpFAC61IoIVWJNBCKxJooRUJtNCKBFpoRQIttCKBFlqRQAutuHsLlke96zLl+x6AQNVLVfzt0t+GPb7/6/u544Y7PKhoBNOAe4AZQD7QDTRjX1J/2JmXlHsKR5ELPQAfvuVhqssGG5ZOLfDJcga3AiuxjwFagGNACDvktyGB9kKmPQAT5xJ0vtFJZFWEQLE7C9OsnbOWFbUrxr39pXcv0XO8h+iqKIbp0BFnEHgYO8wHgG1A/2pqCnDwJnVvAu1y77qJ2npka0Y9ALt+30XLCy20/riVKeunUPa1MseD/er+V9l1YtfA+KVlIy2dP+jkiydp+782mp5t4tM/+rQzwZ4B9N8w/ycGwwz2fRgO3vjkTaD90S57TG+efHPI+LGVj439pABYMYu2X7TR9ss2pv1qGoX3FDpUIbxx7I0h4xUPjW+27jvbx7F1x2j6fhNz35tL3qeyuJjOlb9uf5/HB7CPp/ttzN7LXcmbQLvcu26iXvnsK9zWcBtW/PI9U2P8IfYd7bPfZlNAAMyoSWCKszP0azNe49af3Tr4wGdH3777YLf9g8JeE6/WgTXxYlf8XAK0ASeBD4B/yO5LXU2OoUdhTjOp2lE17u0v/s9Fmv+1meDMINF/i1K4tBClnL03r3RBKfO/Mn/c2x/4wgEuvHGB0iWlVP+4muI7i7Nf1CnsMxoFwL3A77A/FHYggc4lkS9FCM0Okb8g3/EgT9Qtm26hb2MfxXMcCHK/BLAdeBT789KNwGkg4txL9pNAZ5FRYFCw0N/Lh4UqQ4QqXVjY8f+xZ+SF2B8S78A+FPkIOOjcy8oyBi6SZQyyS5YxENqTQAutSKCFViTQQisSaKEVCbTQigRaaEUCLbQigRZakUALrUighVYk0EIrEmihFQm00IoEWmglw+uhVSdw1LlyJq0CaPW6iDFIjdlxs2VZwxYhyfSOlaOWZc3NUkFZp5Ta6+f6QGp0mhxyCK1IoIVWMg30fzpSRfb4vT6QGh2V0YdCIfxODjmEViTQQisSaKEVCbTQigRaaOXvDfHJlPquMXIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.117143Z",
          "start_time": "2018-04-02T13:43:05.563Z"
        },
        "id": "7PQ2TG6BsYTl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 220
        },
        "outputId": "dd0fd34e-0994-4e6b-bba8-4445bda63624"
      },
      "source": [
        "from IPython.display import clear_output\n",
        "from time import sleep\n",
        "mdp = FrozenLakeEnv(map_name='8x8', slip_chance=0.1)\n",
        "state_values = {s: 0 for s in mdp.get_all_states()}\n",
        "\n",
        "for i in range(30):\n",
        "    clear_output(True)\n",
        "    print(\"after iteration %i\" % i)\n",
        "    state_values = value_iteration(mdp, state_values, num_iter=1)\n",
        "    draw_policy(mdp, state_values)\n",
        "    sleep(0.5)\n",
        "# please ignore iter 0 at each step"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "after iteration 29\n",
            "iter    0   |   diff: 0.00000   |   V(start): 0.198 \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAAA9zQYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO2deXxU1f3337MlM9n3lZCVJYQl7CDBoIi4sIi2rrVP1WrFPnV9fF6Vtv5Q+lJf/T1u/VmXKuoPS8X6a6170YKKgECAyBJIQtgSyB6SyTbJZGbu88c1MwwTkrkzZyAM9/168WLOvTOfOZP53jPnnnM+56uRJAkVlWBBe74roKIiEjWgVYIKNaBVggo1oFWCCjWgVYIK/VBP0Gg09wD3yKXwqTA2wFVSURkao7EMi8WiOfO4RsmwnUYzTZoyRcww3+7duwCYNWu2EL1t274DoLh4nhA9gG+++RqAhQuvEqK3fv2/ALjuumVC9AD++c8PALjllluF6L377l8B+PnP7xai98YbrwPwwAMPCtHr591336WhocEjoNUuh0pQoQa0SlChBrRKUKEGtEpQMeQoh7f0hvVyouAEXXFd2PV29FY9pg4TGXszCO0OVay3u3g3VpPV4/iELRMI7wj3qY7bZm2j19TrcXxqyVQiOiMU630z+Rt6jD0ex2fvmU1Ud5RivfXj1mMJsXgcn1cxjxhLjGI9gI9yPqLL0OVx/KpjVxHbG6tYb13KOjr1nR7HlzUsI74v3qc6vhn1Jh26Do/jt7bfSqI9UZGWsIA+Mv0IlmgLkU2RhHaFYjVa6YzvpM/Y51NA9xPTGIOx2+gsG6wGv+sa1xyHyWISppl4KhFTr0svxBbil16yOZlwq+uiDbX5/vfrJ60zjQir66INtfunmWHJIMrmumiNduMgz/aObGs20Y5oZ9nkMA3y7IEREtA2gw1LtAWdVUfed3lokEdTHFoHnDawopE0SBplw35JJ5KIa4wb8JwvegCpdakkNCd4nuiX8hgMGpz0xnSSW5MHPikp18s8lUmaOU2YHkCuOZcRnSOEaY7pGkNWT5byigxCgbWA3L5cvzSEBLTOpkNr02IPsVNeXE5yfTJxTXEk1Seht8tvMal1EnOa5vBF6hdsS9jmtXbjiEZ6o3vRSToAJuydAIDRZuSO8js4GHuQLzK+UFTfxpRGrOGu7kxhWSEACw8vJKY3hi9yvqAivsJrvZNJJ7GYLM4LeVrFNABSzalcUX4FO0fuZHv2dq/1jscdx2w0O/92sw7Pcp5btnUZLZEtbB6/maaYJq81j0Qdwaw1O8tF1UUA5Fflk1OdQ8mkEiqzK73WqwivoD6k3tmgzDb7P59QFlLGCf0JZ7nYUqxYQ0hAayQNmd9nUj2pGku0hWPRxzg25hixnbE89denGFvrml1cdmIZdaY6dnip3ZbURltSm7O8+i+r3c5PbJmITtLxuYL6NiU20ZToCoY/r/mz87GExE1lN/FU0VPe68U10RTn0nvlrVfczl9y9BI6Qzv5yEu9hugGGqIbnOWX3njJ7byp18SVu65k7fy1XtfxZORJTkaedJZffO1Ft/PFO4rdPsNQ1JhqqDHVOMsiAvpoyFH3Op2vgAaIrY0luj6azvhOOuM7ac5spjWilV8v+TW5O3IZax7LFfVX8FnaZxyJPOK17ujdo926HI/NegyAEHsIdx68k/1x+9mWsg12/MprzYJ9BW5djpXFKwG4/MjlxFvi2ZC9AZvO5rVeYXmhW5fjmYXPAJDUnsRVZVexLWcblUmVUO6d3oyjM9y6HC9e5wq+pVuX0hTdxK5Ru7yuH8Dck3Pduhx/vkW+iPOO5jHmyBh2FO7gVOwpr/WWf7ucGzfcyNqfe39RDcWqdauwTrXSntDus4aQgJY0El2xXUSciiCqKYqopij0Vj0nxp/ArrcDUB5dTnm0l9+oF1h1Vl4d/6owPYCNORuF6jVGNbJm9hqhmh9e8qFQvarsKqqyq4Rqnk+EBLRD66CyqBJjhxGT2YTWrqUtRe4mRDUpH75SUfEVIQGtdWhJOpxER0IH7UntOHQOQnpCSDyWSHLVWe7+VVQCgLCbwhFlZxkS8pEp30wRqgcwa9usoZ+kgOJS5Tctg7HwwEKhegBLjiwRqndz/c0ATD45WZjmne13AlBUXsTGqf51+9SpbxXFmLpMjCofBcCEXROEaM7+QB4lmfOPOa75AB9QA1pFMb3GXnqMPdi1dloSW4RotiW1IWkk2hLbfJo46kcNaBXFOHQOds3aRUNqA7UZtUI0q6ZWYU4wU1ZU5peO4j60Viv2GhjuegB6vbDhegCMRv/XPZxJWFiYUL2IiMEXazUXNtNc2EwE3i3qiooaYrQrCnbfu/uHh0OPjOl0ugGPK7ZgwU6vn6+iEihSU9Oora1VbsHSaDT3aDSanRqNZid4PzWqonI+UNxCT5vmR4/9NHbuLAHgkkvmCNHbunULAJdddrkQPYCvvpKHkK69dpEQvU8//QSAm2++RYgewLp17wJw110/F6K3evUbADz44ENC9F544XkAfve7x4Xo9fPGG2/41kKrqFxIqAGtElSoAa0SVAgbj9p7+V6sYZ4ewHGbxhHWrnxIadfcXQP6/yZ9N8lnT+F3M78b0AM4bec0IrsiFettnLgRS6inB7BofxHRlugBXjE4H+V+RLeh2+P4wqMLffL/AbyX+t6AHsDr6q/zyQO4Omo1HdoB/H8dt5JkT/Kpjn/U/hGzxuxx/G773aSQokhL7AArEN0QTWiXy6+mt/r3FrFNsW6eQn/1AOJb4t08hSF9/nkAk9qSCOtxXbT+egBF+/9gAA+gw7+x8Ow+d/9fmMP/cfBR0ihiJdeFG4ZyTeEBnVCdQGyDb63JQCSdSCK+yTc38dlIrUslsUWZm3gwMpoySGlT1pIMRk5bztn9fz4yums0WZYsYXoF1gLy+vKE6QEUOgoZ6+feicIDunlkMx3xrp+kkQdGApDWncakU5PYlLxpQFv92Wgc0Uh7nMvBkF2RDYDOoeOK6is4GHeQ6qhqRXWsS62jLcZl6xp1WF5oU1BfQJgtjN2pu7Hr7F7r1STW0BLpWtNQUFMAQKQlkqlHp1KaVYo5zPMn9WwciTlCY1ijszyl0bXycMr+KbTEtnA87biiNQ+V4ZXUh9Y7y7Pa5JWHiXWJpNamcnD8QfpC+7zWO9P/N88yz/vKnIXvtd9zXDruLC+UlK8+FB7Q5mT3L+69Ne+5lS9tuJSXxr7k9Xxja2KrW/mvb/3VrTyndg47UnawVUEdW+LdF9S888Y7zscSEsXHinn+kue91muMaXQrv/XqW27lKUen8OnkT/nUS73aCPf1Ea/98TW3sl1jpy6xjk8v91YRN/8fwEvPuvsUx+0bxyfXf+K13lGDu/9PREAf0hxyu0gX2odBQBdsK+DWXbeileQBlH6H9/i28UTYIiiLKaMtpG0wCTfGlI5h8Z7FJFrkLsKOZNlea7QZmdgyEXOomYNxB6F+MBV3Li25lCWlrnXCu9Jkf97U2qnYNXZKU0pxaBxe682omMGPtv/IuR/HnpF7AEhpSyG5PZmTcSdpjGocTMKNohNFFJcVk9Yo+woP5B5wnht3eBxWg1WRQxvg2pprWbJlCRpJjphDYw4BkF2Vjd6u53j2cXqNnjfhZ+NXW3/F9V9cz4cPirOErVq3CsOlBrqSvf8FPxPhAd2j6+GDzA88jn8gfUCUNYq2UO+DuZ+taQO3v+t71mMONSvem+OU6RSfjvFs3f6d+2+0khaLwXPkYjDsWjvf5H/jeUKCaEu0ou5GP4czD3M487DH8V0Fu+gJ7cGh8/6CA+gx9bBjjqfXvmR2CaE9oVjClX3m4YrwgD4bDo3Dp2AejDajWL1evfctlFdo8CmYB6M7zHNYzx8cOkfQBDOoEysqQYawFnrixomipACY+u1UoXoAs7eLyRbQz+V7xS2EAlhyWKz/D+CmupuE6t3VfhcAk2onCdO833E/IHsKt1/q/Q5TA6G20CqK0dq0ZBzIACC+RswcQWppKgAjdvo3/q4GtIpiQnpC0Dlkx0j8STEBnbxf3u4ioSJBNcmqnFt6Ino4Pu44Nr2NI5O939ZtMA7Pl0d0Dl9+2C+T7Dkb5VAJLvYX76diZgW2UO/3AByMjrQOtt27DUu8fyMuqqdQ5YJkxIgMampqVE+hSnCjuIUuLPR+0c5gfP99KQDTpk0Xoifaowgun6JfnTo35L/1T3/6vwTpwZo1/w3A8uX3CdF75ZWXAXj88f8Qovfkk08A8OyzzwnR6+f555/3rYVWUbmQUANaJahQA1olqFADWiWoEDYOXXZlGX1hno6H0V+NJsys3Bsm2nQLgo23DwIxwDpcuVOygJ8BPcAzyuv394y/D+jmWXRiEXHWgVPbDcVfEv4yYFLLH7f8mATbAKnthuBFzYsDGlrvcdyj2NDaz+87f0+r1Opx/OGwh0nXpSvSEj6xElUfRUiXy3Sq7/XvLUSbbiEwxluRjOgaQaTN5UIXkdQyszdTqEl2lDSKOFwXmS+G1jMZpxtHvNY1lR6hUZ7dV/g3GXc8jpg6zzS+WklLrDWWllBl+wmLNt1CYIy3IsnryGNk98gBz4V3htNj6lHkeQTIt+ST3ZvtcVxr02K0GOmOVLbOerI02W9D65nMMMxggsG/DdSFB3R7RjvpIenOJJRLNy0FYLx5PEaHkX1R+3g/432v9ZozmrHGWunVyl2FftOtP5xKPTWg8dYnJiN3NQAvdoH1iqrIKnp6ewjvkrtBN++82Xku70gevSG9lEwp4dDoQ15rVuorsde7LoJbd98KQFZlFgDH846z+9LdXuuVakqpsdbgCJGdM74YWs+kxFLCYbvLpXOd8TrFGsID+lTaKbakbXGWV324yu382I6xRPZ5v6mLOcWMGVefTURAN6W4z3j6FdBj/KzMAJwIP8GJcJej+rd/+63beb1NT/bxbEUBfSTiCEfGuhYSrVi3wu18+rF09s/c77XeIc0hDoW63l9EQJdpyuC027BhEdBZ27PcuhwPFcq7WKZaUpnYNpFvE7+lW+/9z9uSL5fw0JaHeHTao8LquGrdKqqlasri/dstHhj4ptBP5tXPc+tyvPVTl4t8/L7xtMa1cjLt5EAvPStXtV3l1uX42/K/ARBfF09SbRKHJhzCFuL9QqMHdzzI0s+W8tXjXymqx2CsWreKuEVx9KR67m7lLefsbqjOVEedqe5cvV3Qsn+C962oN7SkttCSKiZPynBAHYdWCSrUgFYJKoR1OQq+KBAlBbhMt9OqpwEQZgtT1PceiKnfTiXcGk5ReRGfZnm/69CAvDDAsWPASt8lb6i5wfcXn4WfNP9EqN4D0gMgwcQK+fvR9eiwG/1bgfnbiN9iMBuYUD6BqjlVfvWhh3ULbbKZmNs4F4Al1WIc0TdUyUFz7bFr0TuG14TKhYKx1Uj8YXkcP2tTlhDNnLdyAMh+JxuU7aHjxrAOaIveQlVkFQ4clCSUCNHcmbwTCYn9cfuxacXYhy42emJ66EzsRNJI1E9UsAfbIDTPakZC4tTUU35F5bBvoj7M+JBJrZM4HOm5LZYvHIw7yOa0zc498lR8QAMViyuIORZDV4rv+9CdTsv0Foz1Rhoub/CvaqqnUOVCJCsrm6NHj6qeQpXg5rznKSwqmitEb/PmbwG44ooFQvQA/v3vLwFYsmSpEL2PPpK3nr3jjjuF6AG89dabPzwS63t87jnv98cejIcflmeK3377v4Xo9bNy5UrfWmgVlQsJNaBVggo1oFWCCjWgVYKKYZt4s2ROyYD+v8JthUR0KrfmAGyetnnAxJszS2f6lHjzy4IvB0y8WXyw2KfEm++nvz9gkswltUt8SpIZCN/jqo5VA/r/Hgl/RLH/z/namkdosXuu+Hsi9QkyQzMVaV0QiTdPT5Jp6DP4pQeQcCpBqGayOZnwXpfJtj95kK9kdGe4ewr99P8FgnF6d/9fuMa37L6nM8k0iSS9KxttlE65BWjYJ95MqU0R7v9Lq08j6ZRvaXwHYmTzSFLNqcL0RnWOItOirGU618w0zPTb/3cml0ZcytRw/zI3nLPEm/lt+SyoXcDnIz7nUKT31qH6tHrMsS4LVk6lvIgl1BbKz/b/jAPxB9iWtk1RHWtTammNdv1sjjkq+6guO3QZ8d3xfJX3FS3h3i96r06odku8Of7EeAASzYlcWXolJaNKOJTm/Wc+FHGIeqNrjcTM1plev3ZAAuB73N63nSp7lbO8zLgMAH27ntzXc2mZ3kLLbGXGgU2dmyjvKXeWb4u/TXG9znnizXsq7+HV0a8qSrzZiiv41r651u18Wmca6Z3pKDECNcc1u5XXvL7G+VhCYnTTaP5w2R+81muIdl9/sPrl1W7la3ddy1e9X+FtRr+aMPckmX4HdAB8jwdsB9zKT6540q1sOmnC1GBCCXsse9zKwyKgR+0YRU51jrP8zHj5zmNC6wSKG4pZn7aeYxHHvNbL35NP1oks51LP56bKu1ga7UZ+tv9nlMeVs3HkRmj5pdeaU/ZPIafWVcc/XfInABZULiDWEsuGURsUrcSbXjWdsXUuS/+b8+XZu+S2ZObvmc/u3N0cyDgAXrqnrjl+DY++9Shr7lgz9JO9IQC+xztMdzClcwoauzxZV7ZC9mcazAZy3syhdUor9Qvq4TvvNVetW0XObTn0ZXifovlMhAe0Q+Ogxej5U/N16td8nfK1TzO0HaGeO/8APD3zaZ/0bFobbWGeOQ7fn/TD9gpKNTVgDvfcTcgcbqYyrVLcrPQwoy/WM/CsCVb2/X7fefvM53b5qOgPOdz1AqU53DmPn1mdWFEJKoZt4s3pW8Ts7H86RTuLhOotKBO3sg/gxyd/DEBEu28TRx4EwPf4u8jf+f7is/BsxrMAZJZnUkutX1pqCz3ckGDuJnlJ7SXfXnKeK3PuiHtL3vgx4fWE4PUUXpRoAEkePtRIF1EH3CF/ZgnJrz64GtDDkF3Td2HT2/h+8vfnuyrnDPN1ZiSjRNvNbWrizWCjMaWRd29/93xX45xij7dT82rN0E8cAtUkq3JBkpc3ikOHDqkmWZXgRnELPWmSmM1Z9uyR+4cXY+LNRx75P4L04Nln/x8ATz65aohnesfjj8vDcq+++poQvXvv/QUAH3/8iRC9fh566CHfWmgVlQsJNaBVggo1oFWCCmHDdgeuPEBf+AB5CjeOxmRWti4WLs48ha+Hv067tt3j+O1dt5Pk8M1h85zjOdrwXFm4XLOcVI1yl82KxhWccpzyOP6b+N+QYcjwqY537buLRmujx/EX818kJyxngFecHfF5CuvUPIX+kmPLIcbhylNjkpQ3CGcymtFueQXD8c8DOCF0Aom6RGc5Quv/+pPp0dNJDXVdZNF65UbjgOQpjK5TXpGzcTHmKRzfN55RtlFCNadqppKvyRemN8c0h0JjoTA9gAUJC5gdM9svDfFp3TJP0ZngsuKn75Ot7dHWaPI68yiNLcWhOWP1iQT55nzaDG3UhbsnFjqbR3EoYntiyWjPYG/iXo9Rt8YRjcM6T+F+w35O6Fxp3S7rvcz5OKUyhc74TjrjPbc7GIxd0i6OSked5Wu01wBgbDESURdB87hmRXdUWyxbqLRWOss3Rt0oP7BB1I4ousZ3YY9StrP/l81fsr/DZeu5O+NuRa+HQCTeTHXvA77/F/ckm0vrlvJS7kvOcqg9lPsP3E9Sr3sf8eaim2kIa/DwKHoT0AuOL2BO7RwMkoGbqm5yac64mQZTA62J7vtKDLc8hUf0R9zKzz71rFvZrrNzMv8kexfu9Vqzkkq38tOPP+1W7onuYc/P3T19g7Gvd59b+Xf/131ZqcPgoPHHnv3iwSgxu29qPywCunBTIY9teYxQu9zvNevlgIy2yd2QLl0XNo1rckZCoj2k3RnQ7YZ253GAX7//a6qt1VRGu38hg9Fl6ELSSCBBt74bu8bupvnTz3/KYe1h+Tn+EgC/3pLuJSz6ZhEZ++WbrJ5w1+Y4xi65798b7nlzOxh3nLqDG9fciNYmN8PWCPmGO6RTvt/pC+9D0nn/91gesZxr1lyD8ZhcH1uU/J3q2+WQkvQS9nBlLfSqdauY8sspOLJ8Xz8qPKDbQtp4auxTHsdNdhPplnSqwqvcugBWnZXXxrzGiK4RdOo7aQttc+oAvJ/1vuI+9Na0rexO2k1qVypHo10/s+ZQ+eLanrJ9WPeh0UBFUQUVRRUep2JPxtId0604oHvieyh52DOth6HTgKnFRHum5+jKYEh6iZP3DZD80wFhB8OwjLIghUjwuiJZvzlnt/cWnYWqiKqznj89FbAIevQ9bsEcLLSme27D5Q99EX30RfjusvZAC90F/mUr8/PtVVSCB2Et9LgvxomSAsR7FEHOUyiMAPj17u5SfhM0FA9rHxaq91SSZ3fSX1ZPkDfmiVgZQTf+te5qC60yLNDt08n/79b5paMGtMqwIORtebQl5L0Q1SSrcuFjvU0eRrTeaA3uxJsqFwf2yXYsKyzYJ/uXN1z1FKpckOTnj+PAgQOqp1AluFHcQo8dKya3c3n5wX5VIXr9fj2xOwXKmr/4xb1C1F577VUAHntshRA9gKeflofRXnzxj0L0Hnjg/h8eif1edu7cJUhP5vbbb/ethVZRuZBQA1olqFADWiWoUANaJagQNg5dtbgKW7jnJjRZ/8rC2KYwz14ADKiB0FybuHbARJk3NN1Agi1Bsd7LIS9j1nimtrjTeifJUrLyCgJPmJ8Y0NT6aOSjjNCPUCYWiO8FWLxhMXWWOo/ja+euZUy0MgeF8ImV8JPhzkXjALpe/+bmLwRG9owkyu7yX5kc/pla8+x5xEguk2yY5JvL/XQKDAUkaF0XmQhTq2jmJs1lRLjrIosNUe4l9Tmg9Q79gJmiYo7EEHlSeZrhC5mx3WPJ7h3YxqW1aXHolS1OmOSYxGjH6LPr6RyKR9VmhcxiYoj4FYwiWTpyKfNS5vmloTigY/tiWdq0lJntM/k44WPqQuSfihpHDTZsODIdNCS58vYll/r2UwkExIAaCM3ysHLae9udG5Rfc1Q2oMY3xpO/J5/DYw+zb/q+wSTc2KPdQ52lzrkFxLLaZc5zEz+cSGdCJxVXVNCa6f1i/20926iyuQwW14dd7/VrPQjE9wL889g/2dXiGq9+pOARxRqKA3pa+zRmtM8AYHHzYufxjfaNdNFF3Uj3vpBfAR0AA2ogNKuN1VSnVjvLT/3Zfc1wbnkuLYneZ1Wt0lVRFeUKvif+9ITb+cimSMZsGMO2O73PoFtmL4PTlkn4FdCB+F6Azc2b4bScqOckoL+M+5IToSe4rPUy1iWv41SIfMPRYpC/sPRv08V1OQJgQA2E5pWnrnTrcqz7xToAIlsjmbxtMuWTymlMa4RvvdO7oe8Gty7HF4994Xw8/qPxdCR3UDNF2ebgq9atoiCigJrb/d9UPCDfC3Idr1x5Jbo83++7lPehNXAw4iAHIw4O/dyLnI7YDjZdvUmo5v4lXqajvUhRx6FVggo1oFWCCmHj0Hkf54mSCogBNRCatzXd5vuLB+A+631C9QD+I/o/wAETyyfSVui5C6kiAvG9AB/P/xipT6JzZSd4bjirCLWFvghI+pe8K1XM9zGENIcM8ezzQ/fybuf/ksP3Ha3UgL4I6BrdhUPnoC+yD2usn01ggNDP1oMWdNN1aLS+r8VWPYUXAV15XbRNaaN9fDsM05UIITeHYN9lx/i/Fa77OQM1oC8STvxE7FZrotGEawh72f81K6pJVuWCpLBwMqWlpapJViW4UdxCjxmjbOf4s1FR0T93KtaMefXV1wjSg88//wyAG2+8aYhnesff/vbeD4/EG3n/53/+LkTtRz+64YdHYr8Xs1nZdr1DUVxc7FsLraJyIaEGtEpQoQa0SlAhbNju8OLD2CI8HSyZn2cOD08h8HXh11hCLR7H5+ybQ1S38pXqn+R9QneI537GCw4vILbXh1R0AfjcyyuW09TneTP/n7n/SbZJYbKkAH0vE1ZPoLq92uP4t7d9y8QkZS6bi9JTmNiaSFiPa8wzpM+/6eDUjlQirC6PXn/CpOHE1MippISkOMtReoFWE0FclX0V2TGuiywhTLnRWHhARx+OHvaewoymDJJb/XDSnEFOWw7pHenC9ALB/Nj5zIiacb6rMSi3j7+dRXmL/NIQHtDmXDOWZNfPetJu33JUAwHzrtUk1tAS6bJEjauW02kUHi8kojeCkuwSeg3uWaZi2mOYcnAKe0bvoSXW3U51JOYIjWGunHyTGyb7V8EAfO4NrRso6ypzlu9IvcN3sQB9L+/sf4fNJzY7y8/MU96HER7QXelddOHa0NGvgA6Qd60p1r1P+fZrbzsfS0hMOzaN/5r/X85jI+pHsGjTIjRoGHPcValvHvyG7pBu6iLdfZR+B3QAPveuDvfNEv0K6AB9L/86+i+38rAI6FXrVpF2LI0/ZP7Bf7EAedcu3XMpxQeLneWNYzcCcEnVJQBszd2KXetylLbEtFCeVU7+sXxqkmuoTpFvYPpb8Vu234LGpMGm97wp9okAfO5V61YxPWk6nfcLmBgLoKfw7jfvxlTg+74mAVmcpBE6EyaeTlMnO7M916SUjixFgwabzj0wLUYLX8/4mq2FW7EarM5JNKtBXopZnVY97PvQFwvqarvTsOsGT4dgDRmea4lVXKgTKypBhbAWOvfjXACKKoo4avQzJXGAvGvzvp/nn8AZLKryb4jJgwB87lfGvAJAfHk8vUnK8oN7EKDvZd9d8q5Su1fu9k8IwS10TncOANk92UTahvdY9MWEYZcBgNBNoeA5UTosqH+hHoC6Z+pQsgL0TIQG9OlbyIY5/HcfqIhBVyfP1koaCY1leN6wd26VR1+6S7td6XJ8QGhAl0SW0KpvpTSilIaQhqFfoHJO6FnQgyPcQe8VvUhxfkRLAElfmQ4aSH8yffiYZCWNxJNZT2LTCBqPVRFDKLS91IYUOjyDGcA0zsSEignoE/0LSdVTqHKBokGSJNVTqBLcKG6hJ07sE/LGe/fuAWDatOlC9HbuLAGgqGiuED2AzZv7978V66/7+ONPBOnB4sXy0OG+fWJ2JdbEsGUAAATHSURBVJ0wYTwgzgMYHd2/ekn8zahPLbSKyoWEGtAqQYUa0CpBhRrQKkGFsHHogwsP0hfuecM4asMoTGbl61v3Xr4Xa5jn6rZxm8YR1u7bLGTJnBJ6TZ7rGQq3FRLRqTBvXwAMo3ftu4tGa6PH8RfzXyQnLEe5ILDw84XUdtd6HH9//vuMjRmrWE+kodVJOlAEZAAmoBtoRB4hVpj5RPjy0ci6SEK7XCbR/tRkvhLdEO2uZ/W/yrFNsZgsrovM0GfwW1Mk06Onkxqa6ixH66P91ixOKSYjIsNZjg31wZV+GiIMrQCMA36E3FdoAiqBUOQgn8D5D+i4Y3FE13l+AQaHgeSeZE6EKdsFM6E6gdiGgf/4I7pGUG+qHzABqMFuIMGSQF2EZ8rdlNoU4pviFdXjXLIgYQGzY2YPeE57TIsjwQEKf1CWZS1jfvp8j+OODgeOWgf6McpC4WyGVkmS6N7ZTVhhGBrDEEN1BmARcjDvAz4A+nOUagAfrhHhAd05spM8Qx7aH7rnt264FYAsSxYAR8OO8vbIt73Wa85oJikkicg+efXebV/JaSCMdiNJPUl067r5e+bf3eYvp9RP4eqjV2O0G6kLr3NOxZdPL6fX1Is91s6RmCPOodGcSt9+zoGAGEa/bP6SA6UH0NbKf8P7drtSVegqdUihEtZbrPTd4P2cwD/K/sGWtVucAXP/3vsBsO2T/zaGWQYi/+D9Csk1+9bw2TufYTslv/7hsocB6Kvro6+2D32inpEvj2TQP0oG0N97/AZXMIM8ZO/DPJ7wgG5Jb2FD+gZnecUnK9zOp/SkYHR4v/GMOcXMdynfOcuPffKY2/kQRwip3alux1K6UtA75I+W2uU6p5PkVWeVIyvdnu9XQAfAMFpiLqEkqsQZC79a9yv3J9hBe0jZ/fymjk2Q6yrft9Y9n4ut3IZk8X6Sbf2x9fI9xA8pye9dc697Fdvs9FYOsf46/LTH/elfrkDuT/ez0usqAQEI6MzvMt26HA9MfACA+N548jvy2R63nT6t9y1LbkmuW5fj0WmPAqB1aJndNJuK6Aqajc1Q/3vncz7L/YydKTvJMmdRklqCpJG/qLZQ+a+WvydfXJcjAIbRFbkr3LocnR+7jK36L/U4Mh04RivLH/7CrBfcuxw/bDLaV9mHba8N41Lj0F2E01i7eK17l+Mh+T+HxUHTm03ELI4hdGQo/GYQka7THkcBp4BqYC/g4/3lOfMUtoS2sDl089BP9BKH1sGW5C1nPd8Y3khjuOeIwYWObYHYlYyG0QYMo8XdFGtNWpJ/6eUmPjXIIxphwFzgQ+SbwnaGf0CrqHjQB3wGXI98L5IKnAD8GNRRA1rl/LIfuUWeg3yTWIjcFakCygZ53VkQFtD56/NFSQEwcaOPvzmDMH2LmJV9QEAMo6snrPb9xWdh/dXrher1G1qFUv3DPwGoU98qQYUa0CpBhRrQKkGFGtAqQYVCC5amA6gQ+P4JQPMw1guE5sVYx0B85kxJkhLPPKh0lKNCkqRpgiqERqPZOZz1AqF5MdYxEJ/5bKhdDpWgQg1olaBCaUD/WfD7D3e9QGhejHUMxGceEEU3hSoqwx21y6ESVKgBrRJUqAGtElSoAa0SVKgBrRJU/H+gS31Zu7BPngAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 216x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lFsDsEUIsYT5",
        "colab_type": "text"
      },
      "source": [
        "Massive tests"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.118218Z",
          "start_time": "2018-04-02T13:43:05.568Z"
        },
        "id": "ysWWRFiMsYT5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "5cff6387-714a-480f-8218-ff33fb3d05f2"
      },
      "source": [
        "mdp = FrozenLakeEnv(slip_chance=0)\n",
        "state_values = value_iteration(mdp)\n",
        "\n",
        "total_rewards = []\n",
        "for game_i in range(1000):\n",
        "    s = mdp.reset()\n",
        "    rewards = []\n",
        "    for t in range(100):\n",
        "        s, r, done, _ = mdp.step(\n",
        "            get_optimal_action(mdp, state_values, s, gamma))\n",
        "        rewards.append(r)\n",
        "        if done:\n",
        "            break\n",
        "    total_rewards.append(np.sum(rewards))\n",
        "\n",
        "print(\"average reward: \", np.mean(total_rewards))\n",
        "assert(1.0 <= np.mean(total_rewards) <= 1.0)\n",
        "print(\"Well done!\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "iter    0   |   diff: 1.00000   |   V(start): 0.000 \n",
            "iter    1   |   diff: 0.90000   |   V(start): 0.000 \n",
            "iter    2   |   diff: 0.81000   |   V(start): 0.000 \n",
            "iter    3   |   diff: 0.72900   |   V(start): 0.000 \n",
            "iter    4   |   diff: 0.65610   |   V(start): 0.000 \n",
            "iter    5   |   diff: 0.59049   |   V(start): 0.590 \n",
            "iter    6   |   diff: 0.00000   |   V(start): 0.590 \n",
            "average reward:  1.0\n",
            "Well done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.119075Z",
          "start_time": "2018-04-02T13:43:05.571Z"
        },
        "id": "cGe0G7ECsYUC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "21372dfd-e9cd-4630-9ca8-0cb3c6bfa85a"
      },
      "source": [
        "# Measure agent's average reward\n",
        "mdp = FrozenLakeEnv(slip_chance=0.1)\n",
        "state_values = value_iteration(mdp)\n",
        "\n",
        "total_rewards = []\n",
        "for game_i in range(1000):\n",
        "    s = mdp.reset()\n",
        "    rewards = []\n",
        "    for t in range(100):\n",
        "        s, r, done, _ = mdp.step(\n",
        "            get_optimal_action(mdp, state_values, s, gamma))\n",
        "        rewards.append(r)\n",
        "        if done:\n",
        "            break\n",
        "    total_rewards.append(np.sum(rewards))\n",
        "\n",
        "print(\"average reward: \", np.mean(total_rewards))\n",
        "assert(0.8 <= np.mean(total_rewards) <= 0.95)\n",
        "print(\"Well done!\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "iter    0   |   diff: 0.90000   |   V(start): 0.000 \n",
            "iter    1   |   diff: 0.72900   |   V(start): 0.000 \n",
            "iter    2   |   diff: 0.62330   |   V(start): 0.000 \n",
            "iter    3   |   diff: 0.50487   |   V(start): 0.000 \n",
            "iter    4   |   diff: 0.40894   |   V(start): 0.000 \n",
            "iter    5   |   diff: 0.34868   |   V(start): 0.349 \n",
            "iter    6   |   diff: 0.06529   |   V(start): 0.410 \n",
            "iter    7   |   diff: 0.05832   |   V(start): 0.468 \n",
            "iter    8   |   diff: 0.01139   |   V(start): 0.480 \n",
            "iter    9   |   diff: 0.00764   |   V(start): 0.487 \n",
            "iter   10   |   diff: 0.00164   |   V(start): 0.489 \n",
            "iter   11   |   diff: 0.00094   |   V(start): 0.490 \n",
            "iter   12   |   diff: 0.00022   |   V(start): 0.490 \n",
            "iter   13   |   diff: 0.00011   |   V(start): 0.490 \n",
            "iter   14   |   diff: 0.00003   |   V(start): 0.490 \n",
            "iter   15   |   diff: 0.00001   |   V(start): 0.490 \n",
            "iter   16   |   diff: 0.00000   |   V(start): 0.490 \n",
            "average reward:  0.866\n",
            "Well done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.120316Z",
          "start_time": "2018-04-02T13:43:05.574Z"
        },
        "id": "PX7Ez2RbsYUG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "outputId": "79270921-8457-4766-f336-c07f9412d30b"
      },
      "source": [
        "# Measure agent's average reward\n",
        "mdp = FrozenLakeEnv(slip_chance=0.25)\n",
        "state_values = value_iteration(mdp)\n",
        "\n",
        "total_rewards = []\n",
        "for game_i in range(1000):\n",
        "    s = mdp.reset()\n",
        "    rewards = []\n",
        "    for t in range(100):\n",
        "        s, r, done, _ = mdp.step(\n",
        "            get_optimal_action(mdp, state_values, s, gamma))\n",
        "        rewards.append(r)\n",
        "        if done:\n",
        "            break\n",
        "    total_rewards.append(np.sum(rewards))\n",
        "\n",
        "print(\"average reward: \", np.mean(total_rewards))\n",
        "assert(0.6 <= np.mean(total_rewards) <= 0.7)\n",
        "print(\"Well done!\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "iter    0   |   diff: 0.75000   |   V(start): 0.000 \n",
            "iter    1   |   diff: 0.50625   |   V(start): 0.000 \n",
            "iter    2   |   diff: 0.39867   |   V(start): 0.000 \n",
            "iter    3   |   diff: 0.26910   |   V(start): 0.000 \n",
            "iter    4   |   diff: 0.18164   |   V(start): 0.000 \n",
            "iter    5   |   diff: 0.14013   |   V(start): 0.140 \n",
            "iter    6   |   diff: 0.07028   |   V(start): 0.199 \n",
            "iter    7   |   diff: 0.06030   |   V(start): 0.260 \n",
            "iter    8   |   diff: 0.02594   |   V(start): 0.285 \n",
            "iter    9   |   diff: 0.01918   |   V(start): 0.305 \n",
            "iter   10   |   diff: 0.00858   |   V(start): 0.313 \n",
            "iter   11   |   diff: 0.00560   |   V(start): 0.319 \n",
            "iter   12   |   diff: 0.00260   |   V(start): 0.321 \n",
            "iter   13   |   diff: 0.00159   |   V(start): 0.323 \n",
            "iter   14   |   diff: 0.00076   |   V(start): 0.324 \n",
            "iter   15   |   diff: 0.00045   |   V(start): 0.324 \n",
            "iter   16   |   diff: 0.00022   |   V(start): 0.324 \n",
            "iter   17   |   diff: 0.00012   |   V(start): 0.325 \n",
            "iter   18   |   diff: 0.00006   |   V(start): 0.325 \n",
            "iter   19   |   diff: 0.00003   |   V(start): 0.325 \n",
            "iter   20   |   diff: 0.00002   |   V(start): 0.325 \n",
            "iter   21   |   diff: 0.00001   |   V(start): 0.325 \n",
            "average reward:  0.64\n",
            "Well done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.121544Z",
          "start_time": "2018-04-02T13:43:05.578Z"
        },
        "id": "Xf9YzwHGsYUM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 413
        },
        "outputId": "a4302bd1-dfa2-4a34-9fdf-bc6e06de5155"
      },
      "source": [
        "# Measure agent's average reward\n",
        "mdp = FrozenLakeEnv(slip_chance=0.2, map_name='8x8')\n",
        "state_values = value_iteration(mdp)\n",
        "\n",
        "total_rewards = []\n",
        "for game_i in range(1000):\n",
        "    s = mdp.reset()\n",
        "    rewards = []\n",
        "    for t in range(100):\n",
        "        s, r, done, _ = mdp.step(\n",
        "            get_optimal_action(mdp, state_values, s, gamma))\n",
        "        rewards.append(r)\n",
        "        if done:\n",
        "            break\n",
        "    total_rewards.append(np.sum(rewards))\n",
        "\n",
        "print(\"average reward: \", np.mean(total_rewards))\n",
        "assert(0.6 <= np.mean(total_rewards) <= 0.8)\n",
        "print(\"Well done!\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "iter    0   |   diff: 0.80000   |   V(start): 0.000 \n",
            "iter    1   |   diff: 0.57600   |   V(start): 0.000 \n",
            "iter    2   |   diff: 0.41472   |   V(start): 0.000 \n",
            "iter    3   |   diff: 0.29860   |   V(start): 0.000 \n",
            "iter    4   |   diff: 0.24186   |   V(start): 0.000 \n",
            "iter    5   |   diff: 0.19349   |   V(start): 0.000 \n",
            "iter    6   |   diff: 0.15325   |   V(start): 0.000 \n",
            "iter    7   |   diff: 0.12288   |   V(start): 0.000 \n",
            "iter    8   |   diff: 0.09930   |   V(start): 0.000 \n",
            "iter    9   |   diff: 0.08037   |   V(start): 0.000 \n",
            "iter   10   |   diff: 0.06426   |   V(start): 0.000 \n",
            "iter   11   |   diff: 0.05129   |   V(start): 0.000 \n",
            "iter   12   |   diff: 0.04330   |   V(start): 0.000 \n",
            "iter   13   |   diff: 0.03802   |   V(start): 0.033 \n",
            "iter   14   |   diff: 0.03332   |   V(start): 0.058 \n",
            "iter   15   |   diff: 0.02910   |   V(start): 0.087 \n",
            "iter   16   |   diff: 0.01855   |   V(start): 0.106 \n",
            "iter   17   |   diff: 0.01403   |   V(start): 0.120 \n",
            "iter   18   |   diff: 0.00810   |   V(start): 0.128 \n",
            "iter   19   |   diff: 0.00555   |   V(start): 0.133 \n",
            "iter   20   |   diff: 0.00321   |   V(start): 0.137 \n",
            "iter   21   |   diff: 0.00247   |   V(start): 0.138 \n",
            "iter   22   |   diff: 0.00147   |   V(start): 0.139 \n",
            "iter   23   |   diff: 0.00104   |   V(start): 0.140 \n",
            "iter   24   |   diff: 0.00058   |   V(start): 0.140 \n",
            "iter   25   |   diff: 0.00036   |   V(start): 0.141 \n",
            "iter   26   |   diff: 0.00024   |   V(start): 0.141 \n",
            "iter   27   |   diff: 0.00018   |   V(start): 0.141 \n",
            "iter   28   |   diff: 0.00012   |   V(start): 0.141 \n",
            "iter   29   |   diff: 0.00007   |   V(start): 0.141 \n",
            "iter   30   |   diff: 0.00004   |   V(start): 0.141 \n",
            "iter   31   |   diff: 0.00003   |   V(start): 0.141 \n",
            "iter   32   |   diff: 0.00001   |   V(start): 0.141 \n",
            "iter   33   |   diff: 0.00001   |   V(start): 0.141 \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cv2uRDtbsYUV",
        "colab_type": "text"
      },
      "source": [
        "## Bonus area"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UeW4e754sYUW",
        "colab_type": "text"
      },
      "source": [
        "### Bonus 1 - find an MDP for which value iteration takes long to converge \n",
        "\n",
        "When we ran value iteration on the small frozen lake problem, the last iteration where an action changed was iteration 6--i.e., value iteration computed the optimal policy at iteration 6. Are there any guarantees regarding how many iterations it'll take value iteration to compute the optimal policy? There are no such guarantees without additional assumptions--we can construct the MDP in such a way that the greedy policy will change after arbitrarily many iterations.\n",
        "\n",
        "Your task: define an MDP with at most 3 states and 2 actions, such that when you run value iteration, the optimal action changes at iteration >= 50. Use discount=0.95. (However, note that the discount doesn't matter here--you can construct an appropriate MDP with any discount.)\n",
        "\n",
        "Note: value function must change at least once after iteration >=50, not necessarily change on every iteration till >=50."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.122424Z",
          "start_time": "2018-04-02T13:43:05.582Z"
        },
        "id": "uA9zJnt1sYUW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "transition_probs = {\n",
        "    < YOUR CODE >\n",
        "}\n",
        "rewards = {\n",
        "    < YOUR CODE >\n",
        "}\n",
        "\n",
        "from mdp import MDP\n",
        "from numpy import random\n",
        "mdp = MDP(transition_probs, rewards, initial_state=random.choice(tuple(transition_probs.keys())))\n",
        "# Feel free to change the initial_state"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.123825Z",
          "start_time": "2018-04-02T13:43:05.586Z"
        },
        "id": "fepKV_xRsYUb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "state_values = {s: 0 for s in mdp.get_all_states()}\n",
        "policy = np.array([get_optimal_action(mdp, state_values, state, gamma)\n",
        "                   for state in sorted(mdp.get_all_states())])\n",
        "\n",
        "for i in range(100):\n",
        "    print(\"after iteration %i\" % i)\n",
        "    state_values = value_iteration(mdp, state_values, num_iter=1)\n",
        "\n",
        "    new_policy = np.array([get_optimal_action(mdp, state_values, state, gamma)\n",
        "                           for state in sorted(mdp.get_all_states())])\n",
        "\n",
        "    n_changes = (policy != new_policy).sum()\n",
        "    print(\"N actions changed = %i \\n\" % n_changes)\n",
        "    policy = new_policy\n",
        "\n",
        "# please ignore iter 0 at each step"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SQ6GQ2ZBsYUh",
        "colab_type": "text"
      },
      "source": [
        "### Bonus 2 - Policy Iteration\n",
        "\n",
        "Let's implement exact policy iteration (PI), which has the following pseudocode:\n",
        "\n",
        "---\n",
        "Initialize $\\pi_0$   `// random or fixed action`\n",
        "\n",
        "For $n=0, 1, 2, \\dots$\n",
        "- Compute the state-value function $V^{\\pi_{n}}$\n",
        "- Using $V^{\\pi_{n}}$, compute the state-action-value function $Q^{\\pi_{n}}$\n",
        "- Compute new policy $\\pi_{n+1}(s) = \\operatorname*{argmax}_a Q^{\\pi_{n}}(s,a)$\n",
        "---\n",
        "\n",
        "Unlike VI, policy iteration has to maintain a policy - chosen actions from all states - and estimate $V^{\\pi_{n}}$ based on this policy. It only changes policy once values converged.\n",
        "\n",
        "\n",
        "Below are a few helpers that you may or may not use in your implementation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.125320Z",
          "start_time": "2018-04-02T13:43:05.590Z"
        },
        "id": "McmdCD_LsYUi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "transition_probs = {\n",
        "    's0': {\n",
        "        'a0': {'s0': 0.5, 's2': 0.5},\n",
        "        'a1': {'s2': 1}\n",
        "    },\n",
        "    's1': {\n",
        "        'a0': {'s0': 0.7, 's1': 0.1, 's2': 0.2},\n",
        "        'a1': {'s1': 0.95, 's2': 0.05}\n",
        "    },\n",
        "    's2': {\n",
        "        'a0': {'s0': 0.4, 's1': 0.6},\n",
        "        'a1': {'s0': 0.3, 's1': 0.3, 's2': 0.4}\n",
        "    }\n",
        "}\n",
        "rewards = {\n",
        "    's1': {'a0': {'s0': +5}},\n",
        "    's2': {'a1': {'s0': -1}}\n",
        "}\n",
        "\n",
        "from mdp import MDP\n",
        "mdp = MDP(transition_probs, rewards, initial_state='s0')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ug_--an7sYUq",
        "colab_type": "text"
      },
      "source": [
        "Let's write a function called `compute_vpi` that computes the state-value function $V^{\\pi}$ for an arbitrary policy $\\pi$.\n",
        "\n",
        "Unlike VI, this time you must find the exact solution, not just a single iteration.\n",
        "\n",
        "Recall that $V^{\\pi}$ satisfies the following linear equation:\n",
        "$$V^{\\pi}(s) = \\sum_{s'} P(s,\\pi(s),s')[ R(s,\\pi(s),s') + \\gamma V^{\\pi}(s')]$$\n",
        "\n",
        "You'll have to solve a linear system in your code. (Find an exact solution, e.g., with `np.linalg.solve`.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.126518Z",
          "start_time": "2018-04-02T13:43:05.593Z"
        },
        "id": "9FKp-b5UsYUr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def compute_vpi(mdp, policy, gamma):\n",
        "    \"\"\"\n",
        "    Computes V^pi(s) FOR ALL STATES under given policy.\n",
        "    :param policy: a dict of currently chosen actions {s : a}\n",
        "    :returns: a dict {state : V^pi(state) for all states}\n",
        "    \"\"\"\n",
        "    # YOUR CODE HERE\n",
        "    return < YOUR CODE >"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.127349Z",
          "start_time": "2018-04-02T13:43:05.597Z"
        },
        "id": "-CpSlFZDsYUy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "test_policy = {s: np.random.choice(\n",
        "    mdp.get_possible_actions(s)) for s in mdp.get_all_states()}\n",
        "new_vpi = compute_vpi(mdp, test_policy, gamma)\n",
        "\n",
        "print(new_vpi)\n",
        "\n",
        "assert type(\n",
        "    new_vpi) is dict, \"compute_vpi must return a dict {state : V^pi(state) for all states}\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lPjd_K89sYU3",
        "colab_type": "text"
      },
      "source": [
        "Once we've got new state values, it's time to update our policy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.128415Z",
          "start_time": "2018-04-02T13:43:05.601Z"
        },
        "id": "OJkDLgSssYU3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def compute_new_policy(mdp, vpi, gamma):\n",
        "    \"\"\"\n",
        "    Computes new policy as argmax of state values\n",
        "    :param vpi: a dict {state : V^pi(state) for all states}\n",
        "    :returns: a dict {state : optimal action for all states}\n",
        "    \"\"\"\n",
        "    <YOUR CODE >\n",
        "    return < YOUR CODE >"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.129416Z",
          "start_time": "2018-04-02T13:43:05.604Z"
        },
        "id": "PaDC2rgksYU9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "new_policy = compute_new_policy(mdp, new_vpi, gamma)\n",
        "\n",
        "print(new_policy)\n",
        "\n",
        "assert type(\n",
        "    new_policy) is dict, \"compute_new_policy must return a dict {state : optimal action for all states}\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4lH_rQD-sYVF",
        "colab_type": "text"
      },
      "source": [
        "__Main loop__"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.130183Z",
          "start_time": "2018-04-02T13:43:05.608Z"
        },
        "id": "kj-nlVHFsYVF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def policy_iteration(mdp, policy=None, gamma=0.9, num_iter=1000, min_difference=1e-5):\n",
        "    \"\"\" \n",
        "    Run the policy iteration loop for num_iter iterations or till difference between V(s) is below min_difference.\n",
        "    If policy is not given, initialize it at random.\n",
        "    \"\"\"\n",
        "    < A WHOLE LOT OF YOUR CODE >\n",
        "\n",
        "    return state_values, policy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XuU-k4YKsYVI",
        "colab_type": "text"
      },
      "source": [
        "__Your PI Results__"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2018-04-02T13:43:17.130926Z",
          "start_time": "2018-04-02T13:43:05.612Z"
        },
        "id": "Iy5B5hP-sYVJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "< Compare PI and VI on the MDP from bonus 1, then on small & large FrozenLake >"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}