oemof_tsam.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b9d1a9ca-447b-4764-86a8-1308998a2b13",
   "metadata": {},
   "source": [
    "# Tutorial: Using tsam with oemof.tabular\n",
    "\n",
    "@oemof anniversary meeting 2024\n",
    "\n",
    "by srhbrnds\n",
    "\n",
    "## Useful literature\n",
    "\n",
    "- https://energyinformatics.springeropen.com/articles/10.1186/s42162-022-00208-5 , (Blanke, et. al.: **Time series aggregation for energy system design: review and extension of modelling seasonal storages**, 2022)\n",
    "\n",
    "- https://www.mdpi.com/1996-1073/13/3/641 (Kotzur, et.al: **A Review on Time Series Aggregation Methods for Energy System Models**, 2020).\n",
    "\n",
    "- https://www.sciencedirect.com/science/article/abs/pii/S0306261922004342 (Hoffmann, et. al: **The Pareto-Optimal Temporal Aggregation\n",
    "of Energy System Models**, 2022)\n",
    "\n",
    "- **oemof-solph example**: https://github.com/oemof/oemof-solph/tree/feature/integrate_tsam/examples/tsam\n",
    "\n",
    "- **oemof-tabular example**: https://github.com/oemof/oemof-tabular/tree/features/add-tsam-to-datapackage/src/oemof/tabular/examples/datapackages/dispatch_tsam_without_multi_periods\n",
    "\n",
    "## Limitations\n",
    "- segmentation doesn't work so far\n",
    "- works currently only on experimental branches in oemof.solph and oemof.tabular\n",
    "- python versions 3.9, 3.10\n",
    "\n",
    "## How it works"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3ec5fe5b-ead0-4bb1-98bc-918ad1aef868",
   "metadata": {},
   "source": [
    "### oemof.tabular example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9a3e410-ae32-4cb5-9387-5ea0714632c8",
   "metadata": {},
   "source": [
    "#### Make imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51060811-dd51-4522-bf90-76d197a5f637",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "\n",
    "import pandas as pd\n",
    "import tsam.timeseriesaggregation as tsam\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "import oemof_tsam_helpers as helpers\n",
    "\n",
    "from oemof.solph import EnergySystem, Model, processing\n",
    "\n",
    "# DONT REMOVE THIS LINE!\n",
    "from oemof.tabular import datapackage  # noqa\n",
    "from oemof.tabular.constraint_facades import CONSTRAINT_TYPE_MAP\n",
    "from oemof.tabular.facades import TYPEMAP\n",
    "from oemof.tabular.postprocessing import calculations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8458da2d-c07e-49ed-92df-6effdcf3cc14",
   "metadata": {},
   "source": [
    "#### Set scenario names for origin and target (tsam)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04afea48-4ba7-428c-8946-2158dedb6de6",
   "metadata": {},
   "outputs": [],
   "source": [
    "datapackage_name_origin =\"dispatch\"\n",
    "datapackage_name_tsam= \"dispatch_tsam\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1db99516-8ab0-4692-886f-23be0af2e8ce",
   "metadata": {},
   "source": [
    "#### Specify paths for datapackage origin and target (tsam)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76fac4c6-06d2-42fe-ba65-cc59dcaaabba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify paths to datapackage\n",
    "datapackage_path_origin=helpers.DATAPACKAGE_DIR / datapackage_name_origin\n",
    "datapackage_path_tsam=helpers.DATAPACKAGE_DIR  / datapackage_name_tsam\n",
    "\n",
    "print('Origin path:', datapackage_path_origin)\n",
    "print('Origin path exists?:', datapackage_path_origin.exists())\n",
    "\n",
    "print('Target path:', datapackage_path_tsam)\n",
    "print('Target path exists?:', datapackage_path_tsam.exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88e08ff0-f7f3-429b-9b18-03c1210af214",
   "metadata": {},
   "source": [
    "#### Prepare time series data\n",
    "\n",
    "All timeseries data needs to be stored in one single DataFrame. For defaults check the oemof_tsam_helpers.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc556382-0643-42b5-a720-ef6a668b5f70",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\"\"\"Crawl the sequences csv-files of oemof-tabular data/sequences path and merges them into one single DataFrame.\n",
    "    Parameters\n",
    "    ----------\n",
    "    path : Path\n",
    "        The path object pointing to the datapackage JSON-file.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    profiles : pd.DataFrame\n",
    "        DataFrame that contains all sequence data specified in the\n",
    "        oemof-tabular datapackage.\n",
    "\n",
    "    file_columns : dict\n",
    "        Dictionary containing the file paths of the csv-files in the sequences\n",
    "        path as keys and the column names of each csv-file as values.\"\"\"\n",
    "\n",
    "sequences, sequence_dict = helpers.crawl_sequences_data(path=datapackage_path_origin / \"data\" / \"sequences\")\n",
    "\n",
    "sequences.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c37a071-e311-477f-94d4-c0d750ec01dc",
   "metadata": {},
   "source": [
    "#### Specify and run tsam \n",
    "See also TSAM examples as Jupyter Notebooks: https://github.com/FZJ-IEK3-VSA/tsam/blob/master/examples/aggregation_example.ipynb\n",
    "And for all parameter: https://tsam.readthedocs.io/en/latest/timeseriesaggregationDoc.html#time-series-aggregation-class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e429c9c-b7a7-4c85-92c0-e058a0c3a0d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# aggregate time series by running tsam \n",
    "\n",
    "aggregation = tsam.TimeSeriesAggregation(sequences,\n",
    "    noTypicalPeriods=10,\n",
    "    hoursPerPeriod=24,\n",
    "    sortValues=False,\n",
    "    clusterMethod=\"k_means\",\n",
    "    rescaleClusterPeriods=False,\n",
    "    extremePeriodMethod=\"replace_cluster_center\",\n",
    "    representationMethod=\"durationRepresentation\")\n",
    "\n",
    "aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f8f30cb-b492-48b4-987a-3c68b459552f",
   "metadata": {},
   "source": [
    "#### Prepare oemof-tabular datapackage for tsam"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18233032-ce59-41b8-b277-40ca3cab9753",
   "metadata": {},
   "source": [
    "##### Take tsa_aggregation object and derive tsa_parameters, tsa_sequences, tsa_timeindex.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    tsa_aggregation: tsam.TimeSeriesAggregation\n",
    "        contains all relevant parameters and values as a result of executing\n",
    "        the time series aggregation\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    tsa_sequences: pd.DataFrame\n",
    "        contains typical periods and data of all oemof-tabular sequences\n",
    "    tsa_parameters: pd.DataFrame\n",
    "        contains meta information to solph oemof model using tsam\n",
    "    tsa_timeindex: pd.Index\n",
    "        contains the timeindex of aggregated and is used as index for\n",
    "        seqeuences data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca05e448-81b7-4529-a79b-77a86be6274c",
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregated_sequences, parameters, timeindex = helpers.prepare_oemof_parameters(aggregation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "244c8ca2-cb91-44c3-8405-19dd7e2d1ee4",
   "metadata": {},
   "source": [
    "##### Convert and save aggregated time series dataframe into oemof-tabular sequence files\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    tsa_sequences: pd.DataFrame\n",
    "        contains typical periods and data of all oemof-tabular sequences\n",
    "\n",
    "    tsa_timeindex: pd.Index\n",
    "        contains the timeindex of aggregated and is used as index for\n",
    "        seqeuences data.\n",
    "\n",
    "    file_columns : dict\n",
    "        Dictionary containing the file paths of the csv-files in the sequences\n",
    "        path as keys and the column names of each csv-file as values.\n",
    "\n",
    "    path : Path\n",
    "        The Path object pointing to the oemof-tabular sequences directory\n",
    "        (data/sequences) in which the tsa_profiles will be stored.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1665714f-3455-4a2d-8023-b6fd82d4333e",
   "metadata": {},
   "outputs": [],
   "source": [
    "helpers.convert_tsa_sequences_to_oemof_sequences(aggregated_sequences, timeindex, sequence_dict, path=datapackage_path_tsam / \"data\" / \"sequences\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1834b3cf-f00b-4f48-9db8-a2d74bfb0e85",
   "metadata": {},
   "source": [
    "##### Store tsa_parameters to path.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    tsa_parameters : pd.DataFrame\n",
    "        The path to the origin directory to copy from. Defaults to elements_original_path.\n",
    "    path : Path\n",
    "        The path to the oemof-tabular datapackage data/tsam.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6c378d4-4647-4edc-a0a8-f2556e2c9cf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "helpers.store_tsa_parameter(parameters, path=datapackage_path_tsam / \"data\" / \"tsam\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62fc8df3-de9d-4042-a540-320b5e4e6dfc",
   "metadata": {},
   "source": [
    "##### Create and store periods into oemof-tabular datapackage.\n",
    "\n",
    "This necessary for multi-period optimization in oemof, if no_of_periods=0 function passes None.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    tsa_timeindex: pd.Index\n",
    "        Contains the timeindex of aggregated and is used as index for\n",
    "        seqeuences data.\n",
    "\n",
    "    no_of_periods : int\n",
    "        Number of periods used in oemof NOT in time series aggregation.\n",
    "\n",
    "    timeincrement : int\n",
    "        Timeincrement for each period and timestep to allow for\n",
    "        segmentation.\n",
    "\n",
    "    path : Path, Default: data/periods\n",
    "        The Path object pointing to the tsam oemof-tabular datapackage directory\n",
    "        in which the periods will be stored.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    periods: pd.DataFrame\n",
    "        Dataframe that maps timeindex, periods and timeincrement for\n",
    "        each period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b49a42b-8e68-4595-86ce-aaaff3cbf0ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "helpers.create_oemof_periods_csv(timeindex, path=datapackage_path_tsam / \"data\"/ \"periods\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22b36db2-ed5b-4758-a00c-54b0e475d8a6",
   "metadata": {},
   "source": [
    "##### Copy data from the origin directory to the goal directory if the goal directory does not exist.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    origin_path : Union[str, Path]\n",
    "        The path to the origin directory to copy from. Defaults to elements_original_path.\n",
    "    goal_path : Union[str, Path]\n",
    "        The path to the goal directory to copy to. Defaults to elements_path.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70ad9ca3-847c-45ca-ada6-f624b5ebbc92",
   "metadata": {},
   "outputs": [],
   "source": [
    "helpers.copy_elements_data(origin_path=datapackage_path_origin/ \"data\"/ \"elements\", goal_path=datapackage_path_tsam / \"data\" / \"elements\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5557fb17-84ed-4a1b-a8d5-0ebb5fb64dbb",
   "metadata": {},
   "source": [
    "##### Copy and append datapackage.json\n",
    "\n",
    "from the origin directory to the goal directory if the goal directory does not exist. \n",
    "\n",
    "    \"\"\"\n",
    "    resource= {\n",
    "            \"path\": \"data/tsam/tsa_parameters.csv\",\n",
    "            \"profile\": \"tabular-data-resource\",\n",
    "            \"name\": \"tsa_parameters\",\n",
    "            \"format\": \"csv\",\n",
    "            \"mediatype\": \"text/csv\",\n",
    "            \"encoding\": \"utf-8\",\n",
    "            \"schema\": {\n",
    "                \"fields\": [\n",
    "                    {\n",
    "                        \"name\": \"period\",\n",
    "                        \"type\": \"integer\",\n",
    "                        \"format\": \"default\"\n",
    "                    },\n",
    "                    {\n",
    "                        \"name\": \"timesteps_per_period\",\n",
    "                        \"type\": \"integer\",\n",
    "                        \"format\": \"default\"\n",
    "                    },\n",
    "                    {\n",
    "                        \"name\": \"order\",\n",
    "                        \"type\": \"array\",\n",
    "                        \"format\": \"default\"\n",
    "                    }\n",
    "                ],\n",
    "                \"missingValues\": [\n",
    "                    \"\"\n",
    "                ]\n",
    "            }\n",
    "        }\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b82b5a6-5a57-42e0-98bd-f126cab3cbaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "helpers.copy_and_append_datapackage_json(origin_path= datapackage_path_origin, goal_path=datapackage_path_tsam)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87b91fe2-f385-4dce-bda5-23fac29b2693",
   "metadata": {},
   "source": [
    "#### Create oemof-Model from datapackage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "543e2f3c-3ba2-4345-85ec-bcb21b7e00dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "scenario_name=datapackage_name_tsam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15bdddd9-4c8b-4104-b105-91baa4451078",
   "metadata": {},
   "outputs": [],
   "source": [
    "datapackage_path=Path(Path.cwd(),scenario_name)\n",
    "results_path=Path(datapackage_path,'results')\n",
    "\n",
    "if not results_path.exists():\n",
    "    Path.mkdir(results_path)\n",
    "\n",
    "print(results_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2d9f13b-c351-45a3-bf16-7150da378965",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create energy system object\n",
    "es = EnergySystem.from_datapackage(\n",
    "    os.path.join(datapackage_path, \"datapackage.json\"),\n",
    "    attributemap={},\n",
    "    typemap=TYPEMAP,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4202ef6-ca9f-4baa-b66f-a02c2b963936",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create model from energy system (this is just oemof.solph)\n",
    "m = Model(es)\n",
    "\n",
    "# add constraints from datapackage to the model\n",
    "m.add_constraints_from_datapackage(\n",
    "    os.path.join(datapackage_path, \"datapackage.json\"),\n",
    "    constraint_type_map=CONSTRAINT_TYPE_MAP,\n",
    ")\n",
    "\n",
    "# if you want dual variables / shadow prices uncomment line below\n",
    "# m.receive_duals()\n",
    "\n",
    "startTime = time.time()\n",
    "# select solver 'gurobi', 'cplex', 'glpk' etc\n",
    "m.solve('cbc')\n",
    "executionTime = time.time() - startTime\n",
    "\n",
    "es.params = processing.parameter_as_dict(es)\n",
    "es.results = m.results()\n",
    "\n",
    "# now we use the write results method to write the results in oemof-tabular\n",
    "# format\n",
    "postprocessed_results = calculations.run_postprocessing(es)\n",
    "postprocessed_results.to_csv(os.path.join(results_path, \"results.csv\"))\n",
    "executionTime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34289caa-ec5b-4cf7-a83d-41bc4526f482",
   "metadata": {},
   "source": [
    "#### Compare results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07d07694-0dc9-4bc9-b7d5-dc86567ff420",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_origin=pd.read_csv(datapackage_path_origin / \"results\"/ \"results.csv\")\n",
    "results_tsam= pd.read_csv(datapackage_path_tsam / \"results\" / \"results.csv\")\n",
    "\n",
    "results_origin['var_value_tsam']=results_tsam['var_value']\n",
    "results_origin.drop(columns=['region','type','carrier','tech'], inplace=True)\n",
    "results_origin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92fa1bf8-8293-455e-9758-a7ff4d55f3bc",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}