As air quality expert I would like that the automatic computation of the emissions done by an AI model of the Open Data Hub is also applied to a new type of traffic measurement technology, i.e. cameras

[rcavaliere]

With CISMA and u-hopper we have developed an AI model that reads the traffic data of traffic stations and provides as output the emissions generated by traffic. The model is this: https://github.com/noi-techpark/bdp-elaborations/tree/main/pollution_v2

The model currently considers in its white list only "reliable" traffic measurement stations, in particular the one based on the traditional inductive loops technology. A22 has recently introduced new traffic measurement stations based on cameras. Based on deep analysis made with CISMA we found out that the technology is now much more reliable in the past. Therefore the decision, also shared with the BrennerLEC partners, to extend the computation to these kind of stations.

In the code of the elaboration, it's just needed to change this file: https://github.com/noi-techpark/bdp-elaborations/tree/main/pollution_v2/src/dags/common.py You should in particular extend the filter to stations which have sensor_type = camera

Please also check if some adaptations are also needed in the source code, in particular there: https://github.com/noi-techpark/bdp-elaborations/blob/main/pollution_v2/src/validator/Validator.py

[rcavaliere]

@clezag before start this activity, please do the following:

• upload the new emission factors for 2023 (see attached file), as documented here: https://github.com/noi-techpark/bdp-elaborations/tree/main/pollution_v2

Update "parco circolante"

"Parco circolante" stands for the configuration containing the estimate of the distribution of the types of car moving on the considered road.

The folder pollution_v2/src/pollution_connector/model/input contains a dedicated CSV file for each year (e.g. fc_info_2018.csv) with a default one containing default values in case of processing a year with the corresponding file missing (fc_info.csv).

When processing data for a specific year, the DAG looks for the file fc_.csv: if found, the file is used, otherwise the system will use the default file fc.csv (e.g. with files fc_2022.csv, fc_2023.csv and fc.csv, when processing data for the year 2023 the system will use fc_2023.csv and when processing data for 2020 and/or 2024 the system will use fc.csv. No configuration needs to be changed, just add the updated file, clean already processed records and let the DAGs run.

To update the "parco circolante" for a specific year, just add the corresponding file (and update the default file, if appropriate) and then reset the Airflow cache for the previously computed data.

Clean previously updated data on ODH and run again the corresponding DAGs

fc_info_2023.csv

• delete all the emissions already calculated since 1.1.2023, since the elaborations will then consider the emission factors of 2023

The emission calculation for the cameras should start from 10.07.2023 (the day when the data related to the new cameras where released by A22)

[clezag]

@rcavaliere The import is currently running in testing with all the changes in place.
Over the weekend the docker server unfortunately crashed due to the heavy load, so we don't have all the backfilled data yet.

But it should already be possible to check the sensor type and parameter changes.
In the database I see some camera stations that now have emission measurements.

select s.name, d.json->>'sensor_type', t.cname, m.timestamp from station s
join measurement m on m.station_id  = s.id
join metadata d on d.id = s.meta_data_id 
join type t on t.id = m.type_id 
where t.cname like '%-emissions'
and m.timestamp > to_date('01012023', 'DDMMYYYY')
order by s.stationcode , cname