anonymized_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b7455b30-43a4-4550-97a3-40ba75a4827d",
   "metadata": {},
   "source": [
    "# Create anonymized data\n",
    "\n",
    "For examples and sharing, we create an anonymized dataset.\n",
    "\n",
    "## Data requirements\n",
    "\n",
    "- Sewer mains\n",
    "  * Sewer gravity mains and possibly force mains (as one vector).\n",
    "  * We assume the mains to be already imported in GRASS GIS and we do not export at the end (this can be added)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30cccf7e-136f-4b59-87a1-25fbd8941618",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>Note:</b>\n",
    "    An addon tool for GRASS GIS, <em>v.rast.move</em>, is needed. Download it using <code>g.extension v.rast.move</code>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ff6d9b3-6db0-45c0-92e3-f47230431295",
   "metadata": {},
   "source": [
    "## Software setup \n",
    "\n",
    "We will use a couple of standard Python packages and GRASS GIS.\n",
    "\n",
    "For now, the setup here assumes Linux. Instructions for Windows are available at [GRASS GIS Jupyter notebooks wiki page](https://grasswiki.osgeo.org/wiki/GRASS_GIS_Jupyter_notebooks#Running_a_Jupyter_notebook_locally)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5614b4ee-15f7-492b-95f4-a3e090d11b7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Python standard library and IPython packages we need.\n",
    "import os\n",
    "import subprocess\n",
    "import sys\n",
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "# Ask GRASS GIS where its Python packages are.\n",
    "sys.path.append(\n",
    "    subprocess.check_output([\"grass\", \"--config\", \"python_path\"], text=True).strip()\n",
    ")\n",
    "\n",
    "# Import the GRASS GIS packages we need.\n",
    "import grass.script as gs\n",
    "import grass.jupyter as gj"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fc56720-10f2-4e17-895a-27dedec35260",
   "metadata": {},
   "source": [
    "## Get the data ready\n",
    "\n",
    "This assumes we have usable sewer mains already imported in GRASS GIS as `raleigh_gravity_mains@imported_RaleighSewer` in location `data/ww_covid_2022/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9f3a9ce-dc2a-4999-b684-296c4f6bb28d",
   "metadata": {},
   "outputs": [],
   "source": [
    "grass_project = \"data/ww_covid_2022/\"\n",
    "subproject = \"anonymized_data\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8b03c4d-0806-4724-9934-bc13e6167a3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass -e -c $grass_project/$subproject"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1ecf524-5730-484a-b9c6-5f1e852c0451",
   "metadata": {},
   "outputs": [],
   "source": [
    "session = gj.init(Path(grass_project) / subproject)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17b2b7fc-6bfb-4119-a48c-2f7ad64ac6ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "gs.run_command(\n",
    "    \"v.extract\",\n",
    "    input=\"raleigh_gravity_mains@imported_RaleighSewer\",\n",
    "    where=\"Discharge =  'Smith Creek'\",\n",
    "    output=\"mains_subset\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af3e954-497b-4c98-8982-9c9eb71ab664",
   "metadata": {},
   "source": [
    "## Create dense lines\n",
    "\n",
    "First create continuos polylines (multi-lines) from separate lines for better handling further on and then densify them by splitting long straight lines into multi-lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5cd3b65-1b8c-4779-a199-c8c4e93a14f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "gs.run_command(\n",
    "    \"v.build.polylines\",\n",
    "    input=\"mains_subset\",\n",
    "    output=\"mains_subset_polylines\",\n",
    "    cats=\"first\",\n",
    "    type=\"line,boundary\",\n",
    ")\n",
    "gs.run_command(\n",
    "    \"v.split\",\n",
    "    input=\"mains_subset_polylines\",\n",
    "    layer=\"-1\",\n",
    "    output=\"mains_subset_dense\",\n",
    "    length=75,\n",
    "    units=\"map\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be038fdb-ee49-459b-940b-598a1c28bf1b",
   "metadata": {},
   "source": [
    "## Compute kernel density"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "790f22ee-3e8c-4830-9924-052e6db1d110",
   "metadata": {},
   "outputs": [],
   "source": [
    "gs.run_command(\"g.region\", vector=\"mains_subset_dense\", res=100, flags=\"a\", grow=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0e2a228-5819-4b0c-b9a8-53833fb40ae6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# This would be better without bash, the ASCII round trip and with densified lines.\n",
    "v.out.ascii mains_subset separator=\" \" type=line format=standard | grep -vE \"^L|^ 1\" | grep -v \"[a-zA-Z]\" | v.in.ascii input=- output=mains_points format=point separator=\" \" x=2 y=3 --o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd0aa69f-2d4b-4a3b-a09e-83c4a41faa9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "gs.run_command(\n",
    "    \"v.kernel\",\n",
    "    input=\"mains_points\",\n",
    "    output=\"mains_density\",\n",
    "    radius=1000,\n",
    "    dsize=0.0,\n",
    "    segmax=100.0,\n",
    "    distmax=100.0,\n",
    "    multiplier=1000,\n",
    "    kernel=\"gaussian\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2690a306-7a3b-4620-a4c0-066df09a20b4",
   "metadata": {},
   "source": [
    "## Perform the move"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11469fa0-1996-41b2-b6eb-961c09f16790",
   "metadata": {},
   "outputs": [],
   "source": [
    "gs.mapcalc(\n",
    "    \"mains_density_randomized = 2 * rand(mains_density, 1 / mains_density)\"\n",
    "    \" + 1000 * sin(row() / 100)\"\n",
    "    \" + 100 * sin(col())\",\n",
    "    seed=1,\n",
    ")\n",
    "gs.run_command(\"r.colors\", map=\"mains_density_randomized\", color=\"difference\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13f4eca0-fa49-45be-948f-55e00092c723",
   "metadata": {},
   "outputs": [],
   "source": [
    "gs.run_command(\n",
    "    \"v.rast.move\",\n",
    "    input=\"mains_subset_dense\",\n",
    "    x_raster=\"mains_density_randomized\",\n",
    "    y_raster=\"mains_density_randomized\",\n",
    "    output=\"mains_subset_moved\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49d3da8d-128e-4e31-b0d9-7aeea4f6e0f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "sewer_map = gj.Map(use_region=True, width=900)\n",
    "sewer_map.d_background(color=\"white\")\n",
    "sewer_map.d_vect(\n",
    "    map=\"mains_subset_dense\", color=\"blue\", width=1, legend_label=\"Original\"\n",
    ")\n",
    "sewer_map.d_vect(map=\"mains_subset_moved\", color=\"red\", width=1, legend_label=\"Shifted\")\n",
    "sewer_map.d_barscale(flags=\"n\", at=(65, 4), length=2000)\n",
    "sewer_map.d_legend_vect(at=(85, 13), title=\"Sewer mains\")\n",
    "sewer_map.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ce4361d-aee7-4463-b56c-f338c3b21bea",
   "metadata": {},
   "outputs": [],
   "source": [
    "sewer_map = gj.Map(use_region=True, width=900)\n",
    "sewer_map.d_background(color=\"white\")\n",
    "sewer_map.d_rast(map=\"mains_density_randomized\")\n",
    "sewer_map.d_vect(map=\"mains_subset_moved\", color=\"gray\", width=1, legend_label=\"Mains\")\n",
    "sewer_map.d_legend_vect(at=(70, 10))\n",
    "sewer_map.d_legend(\n",
    "    raster=\"mains_density_randomized\", range=(-100, 100), at=(5, 25, 85, 90)\n",
    ")\n",
    "sewer_map.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "635b46bd-622d-41d9-bee0-4358651226d1",
   "metadata": {},
   "source": [
    "## Attributes\n",
    "\n",
    "During the process, we dropped the attribute table, so now there is no link to the (attribute) database and thus no attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c7474dc-0a0b-4ff9-bba9-160b9349e219",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "v.info -e mains_subset_moved | grep num_dblinks"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}