Refactor scenario builder

[guilpier-code]

What scenario builder is meant to do

A bit of vocabulary :
TS : time series. Contains a list of values for every time mark over the year. Time marks can be hour or day.
BC : binding constraint
RHS : Right Hand Side

Scenario subject : let's call Scenario subject a property of an element of the network that is subject to variations depending on the MC year.
The yearly variation of a scenario subject is what changes regarding a scenario subject when the year changes. The yearly variation can be a TS variation (meaning the TS changes at each year) or a single value variation. For example :

• a particular thermal cluster's available production is a scenario subject, and its variation a TS variation : for each year, a TS must be chosen, meaning a TS number among a set of TS
• the capacity of a specific link is also a scenario subject and has a TS variation.
• the yearly initial level of a reservoir in a particular area is a scenario subject that has a single value variation : for each year, we need to pick the value for he initial level.
• the RHS of a BC is a scenario subject with a TS variation. Note that a BC is considered an element of the network, and that its RHS is a property of a BC, so a particular BC RHS still sticks to definition of a scenario subject.

Note that every Scenario subject has a location in the network :

• A wind or solar production belongs to an area.
• A thermal cluster's available production is located in a cluster. The case of clusters production (thermal or renewable) is special, because they have 2 location coordinates : the cluster and the area containing the cluster.
• A link capacity belongs to a link. In Antares, each area owns a list of all links having this area as origin. To identify a link, you have to give its origin first, and then give an extremity area. So a link capacity has 2 location coordinates : an origin area and an extremity area.
• A binding constraint (BC) belong to a group (ok, not exactly a location, but let's generalize a little bit and put BC groups among locations)

Scenario category : let's call Scenario category a set of scenario subject that belong together, whatever the location they have. Here is the full list of current scenario categories, with their specificity.

• available productions of thermal clusters (TS variation, 2 location coordinates : [area, cluster])
• productions of renewable clusters (TS variation, 2 location coordinates : [area, cluster])
• wind or solar productions (TS variation, 1 location coordinate : area)
• demand or load (TS variation, 1 location coordinate : area)
• initial and final reservoir levels (level value variation, 1 location coordinate : area)
• link capacities (TS variation, 2 location coordinates : [area origin, area extremity])

Scenario rule set : let's call scenario rule set the set of scenarii for every scenario category. Be aware that Antares simulator allows the user to define several scenario rule set, but the user must choose (or activate) one of them for a simulation run. Each scenario rule set has a name. When creating a study, a default rule set is created, wearing the name Default Ruleset. In the following, we focus on a single scenario rule set, the default one or not.

Scenario set : it is a list of scenario rule sets available. The user must select one of them for a run.

Scenario rule set... Scenario set... I know, vocabulary doesn't help, but that's how things are named in GUI and code.

Remarks from the previous definitions :

• A scenario subject belongs to a scenario category and has a location.
• For any scenario subject, a number must be given at every year : either a TS number (integer) or a level value (percentage of reservoir capacity : integer)
• we can see that there is some differences over the scenario categories :
  • locations : we can have 1 or 2 location coordinates
  • locations : can be areas, clusters, BC groups
  • yearly variation : can be a TS variation or a level value variation
  • these variations are one of the reasons why the code around scenario builder is messy (see next sections) : the code around these variations has not been properly designed.

C++ class names

• Sets : class for the Scenario set (list of rule sets)
• Rules : class for a Scenario rule set

How scenario builder currently works

Currently, in Antares, setting the scenario of scenario categories is done by filling associated 2d-matrices, where a dimension is a location and the other is the year.

Example :

• if scenario category is wind production, for a given area (location) and a given year, a TS number must be supplied.
• if scenario category is hydro initial levels, for a given area (location) and a given year, a level value must be supplied
• if scenario category is thermal clusters : this case is special, because this category has 2 location coordinates. Coordinate #1 is currently necessarily an area. To each area is associated a 2d-matrix : for a given cluster (location #2) and a given year, a TS number must be supplied.
• The case of NTCs (link capacities) is similar to the cluster case, as NTCs have 2 location coordinates.

If the scenario builder is not activated, all scenarii (meaning scenarii for all scenario categories) are filled with random numbers. These numbers lay inside bounds of course, bounds depending on the yearly variation kind (TS number or level value). In the code, this is the TimeSeriesNumbers::Generate step.
In case scenario builder is activated, user can define a scenario for any scenario category : she / he can fill some 2d-matrices.
Note that, in the code, the Generate step is run whether the scenario builder is activated or not, and that numbers are overridden by numbers chosen by user if the scenario builder is ON. In the code, this latter step is the currently named the apply step.

About 2d-matrices
It is important to note that each 2d-matrix has actually 2 different variations (or 2 sides) :

• one 2d-matrix belongs to the scenario builder code itself (see rules.h/.cpp). It is used to store the scenario builder input data from file scenariobuilder.dat. It does not have any effect on the simulation as is. To have an effect on simulation, the scenario builder must be ON, and in this case, this 2d-matrix content is copied into another 2d-matrix, its "simulation" counterpart (see next bullet).
• one 2d-matrix is located in class Area or in class Study, depending on the associated scenario category (level and BC groups can be found in the study, the other are located in the areas). This 2d-matrix is taken into account during simulation.

Let's call them the scenario builder 2d-matrices and the simulation 2d-matrices.
In the code, simulation 2d-matrices are called timeseriesNumbers most of the time (except for levels : scenarioHydroLevels). Search this keyword to trace them.

Scenario builder workflow during Antares solver execution :

• Loading areas, links with their content (simulation 2d-matrices are not sized here)

• Reading data from input file scenariobuilder.dat : each line of this file supplies a number in a scenario builder 2d-matrix. That's reason why a line is split into several element :

  • A scenario category : l for load, t for thermal available productions, ntc for link capacities, ...
  • A location : one coordinate most of the time (name of an area, link or group). A second coordinate for clusters productions (name of a cluster)
  • A year
  • A number (TS number or level value, depending on the scenario category)

• Resize all the simulation 2d-matrices (taken into account during simulation).

• The TimeSeriesNumbers::Generate step : filling simulation 2d-matrices with random numbers.

• The apply step : If the scenario builder is ON, then the scenario builder 2d-matrices content is copied into their counterparts, the simulation 2d-matrices, but only the read coefficients (the ones != 0).

• scenario builder 2d-matrices content is copied into other structures (see for example function InitializeTimeSeriesNumbers_And_ThermalClusterProductionCost) in order to be taken into account in simulation.

Scenario builder and GUI
Any scenario category can be modified from the GUI. For any purpose, we indicate how it's done.

• Modifying a scenario builder 2d-matrix associated to a scenario category : the GUI holds a pointer to a scenario builder object (named pRules), object from which it can reach and modify any scenario category 2d-matrix, via a public setter. This setter takes as arguments (as expected) : a location, a year, and a value.
  Any user change from GUI (here the content of a cell in a 2d-matrix) must be saved in order to be taken into account in the simulation. When saving the study, this change is saved into scenariobuilder.dat input file, and the study is ready to be run with the solver (see previous section)
• Changing the network topology : imagine that a user removes/adds an area from/to a study. She then needs to get the scenario builder data in memory and the scenario builder GUI window updated accordingly, even if the study is not saved and reloaded. The choice made in the code to deal with these situations is to : first save the scenario builder data into a temporary file, then remove/add the area, and finally load the data from the temporary file. This is what class ScenarioBuilderUpdater is for, following the RAII idiom : saving tmp file at construction, reading tmp file at destruction.
  It's unclear to me why the choice of save/load scenario builder data was made : I'm not sure this strategy is reaches its purpose or is the best way to fulfill this purpose. Furthermore, no instance of ScenarioBuilderUpdater is created when removing a link, or a cluster : it seems that this strategy was not fully implemented.

Code of scenario builder flaws or inconsistencies

• Scenario builder interface is not handy (see discussion 1576)
• Scenario builder internal design is messy (see discussion 1574)
• Simulation 2d-matrices : some of them are stored in each area, and some other are stored in the study (hydro levels and BC groups), which seems to make no sense, as a scenario subject location is never "the study"
• Scenario builder and GUI : Strategy used to update scenario builder data after change by user is questionnable (see previous section)
• Currently, the scenario builder is held by the class Study (as a data member : scenarioRules). So, any client code that needs to interact with the scenario builder needs to contact the study first.
  As you know, Study is a god class, and in my opinion it should only be used to store raw data from input, and not contain objects built from the input data. And even for this mission, it should be split.
  For now, we plan to keep the scenario builder in the study, because moving it would require more work and it goes out of the scope of our purpose here, but we eventually should try to extract it from the study.
  Main obstacles to do this :
  • The GUI interacts in many places in the code with scenarioRules. As GUI should be removed soon, we don't want to invest in finding out a way to do without the study as a container for the scenario builder in the GUI.
  • In the Antares solver, reading the scenario builder input data (from scenariobuilder.dat) is made by a Study's method (Study::scenarioRulesCreate).
    Once these obstacles away, we could create an instance of the scenario builder in the Application, before running the simulation. To do that, we would have to first extract the block regarding time series numbers (resize + ill with random + apply with scenario builder value) from the simulation (see ISimulation::run()) and move it to the Application. This avoid ISimulation to be dependent on the scenario builder.
• Scenario builder is dependent to the Study : not only the scenario builder is nested in the Study (see previous bullet), but it's also dependent on the Study. At least it prevents the scenario builder from being tested in isolation (see discussion 1593).

What do we want

These are general purposes :

• Make the code more clear and understandable
• Design code for change, the first expected change being the addition of new scenario categories
• Design in such a way the scenario builder can be tested in isolation

To discuss enhancements related to particular code flaws or inconsistency, please follow the links in the previous section.

[JasonMarechal25]

I think at this point we could try to answer two questions ?

• What does Scenario Builder do?
• What are we trying to achieve? (independently of what scenario builder currently does)

Those two questions could help us know if Scenario Builder does what is supposed to do and if it does it well. Answering this should help us see the potential faults in the design.

[guilpier-code]

The discussion introduction was modified to take your suggestions into account