Generate global and regional mif

[fbenke-pik]

Following a discussion with @LaviniaBaumstark , we are suggesting the following solution to issue #194

From now on, we create 3 historical.mif files: in addition to the one with any regional resolution, we also have on with only regional resolution and one with only global resolution. These have to be stitched together again in remind2 compareScenarios.

Use cases demonstrated in this PR:

1. IEA ETP data for Industy Production of Steel and Cement is accurate on global level, but not on regional level due to lacking regional granularity of the source. With the proposed solution, we can now remove the regional data in mrremind without losing the global data.
2. The majority of IEA WEO 2021 data is only available for region GLO and regional disaggregation is too inaccurate. Therefore only the GLO data is used and goes into global historical.mif. A small subset of the data is also available for some of the regions, this goes into the regional historical.mif.

@LaviniaBaumstark Is this implementation what you had in mind?

@0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q Do you think this is a feasible solution for your problem with IEA ETP data? Advantage over filtering in remind2 is that we define in one central place which regional data we do not want to export instead of doing it in all the places where the mif files are read in. Disadvantage is that we need to read in three files now.

[LaviniaBaumstark]

I would leave the calcHistorical for all data but the data for which we do not have meaningful regional data. For those I would use your new calcHistoricalGlobal. But I would not calculate an additional file containing the regional data. They are anyway meaningless. Otherwise we could keep/add them in the default calcHistorical

[fbenke-pik]

@LaviniaBaumstark If I understand you correctly, you don't see the need for calcHistoricalRegion. Right now, I am not sure how to avoid using three files.

Using the example of IEA_ETP data, if I moved it back into calcHistorical, we would get duplicate data in historical.mif and historical_global.mif, i.e. the IEA ETP data for region GLO would be in both files. When merging the mifs to run compareScenarios, you would have to take care of such duplicates. If I have to write logic to filter out data from the combined mifs later on, we are almost back to where we are right now (the improvement being that you might be able to apply more general rules to filter data at least instead of model specific filtering rules as we have to do right now)

Btw, I can think of cases where the two files will have global data for the same model and the data would differ. Then you have to write a rule: if the global file has data for that model+region, ignore the global data for model+region in the general historical.mif.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Do you think this is a feasible solution for your problem with IEA ETP data? Advantage over filtering in remind2 is that we define in one central place which regional data we do not want to export instead of doing it in all the places where the mif files are read in. Disadvantage is that we need to read in three files now.

It would "solve" the problem alright, but that is solved already.
Looking forward I would argue this is just adding more convolutions to a
labyrinthian system.

The root cause is madrats pointless insistence on full regional detail, when
there is none. I'd rather that would be addressed.

Regarding this merge request, I'm more opposed than in favour. It does
something I currently don't need, but it will make everything more complicated
over time.

[fbenke-pik]

After talking again to Lavinia, we found a solution that does not require more than one historical.mif while filtering the regional data we do not want to include using the append attribute of calcOutput. This implementation is more in line with the magclass counterpart .