Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making Filter-docs nicer #70

Merged
merged 11 commits into from
Nov 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,20 @@ pyaro.timeseries - User API
pyaro.timeseries.filters - Filters
^^^^^^^^^^^^

.. automodule:: pyaro.timeseries.Filter
:members: FilterFactory, FilterCollection
:undoc-members:
:imported-members:
:show-inheritance:

.. image:: pics/Filter.svg
:alt: Filter UML diagram

.. automodule:: pyaro.timeseries.Filter
:members: FilterFactory, FilterCollections, Filter, StationFilter, VariableNameFilter, CountryFilter, BoundingBoxFilter, DuplicateFilter, FlagFilter, TimeBoundsFilter, TimeResolutionFilter, TimeVariableStationFilter, AltitudeFilter, RelativeAltitudeFilter, ValleyFloorRelativeAltitudeFilter
:members: StationFilter, VariableNameFilter, CountryFilter, BoundingBoxFilter, DuplicateFilter, FlagFilter, TimeBoundsFilter, TimeResolutionFilter, TimeVariableStationFilter, AltitudeFilter, RelativeAltitudeFilter, ValleyFloorRelativeAltitudeFilter
:undoc-members:
:imported-members:
:show-inheritance:


pyaro.timeseries - Dev API
Expand All @@ -55,9 +62,10 @@ pyaro.timeseries - Dev API
:private-members:

.. automodule:: pyaro.timeseries.Filter
:members: DataIndexFilter
:members: Filter, DataIndexFilter
:undoc-members:
:imported-members:
:show-inheritance:


csvreader for timeseries
Expand Down
4 changes: 2 additions & 2 deletions docs/how-to-add-new-reader.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ with :py:meth:`~pyaro.list_timeseries_readers()`.
TimeseriesEngine/Reader subclassing
+++++++++++++++++++++++++++++++++++

It is strongly advised to use the use the helper classes :py:class:`~pyaro.timeseries.AutoFilterReaderEngine.AutoFilterEngine`
It is strongly advised to use the use the helper classes :py:class:`~pyaro.timeseries.AutoFilterReaderEngine.AutoFilterEngine`
and :py:class:`~pyaro.timeseries.AutoFilterReaderEngine.AutoFilterReader` to implement a Engine and a Reader since Filters will
automatically be handled then.

Expand All @@ -42,7 +42,7 @@ The ``YourReader`` should extend :py:class:`~pyaro.timeseries.AutoFilterReaderEn

- the ``__init__`` method of :py:class:`~pyaro.timeseries.Reader` with two fixed args (`self` and `filename_or_obj_or_url`) and several kwargs,
one of them should be `filters`
- it must store the `filters` calling `self._set_filters(filters)`
- it must store the `filters` calling `self._set_filters(filters)`
- the :py:meth:`~pyaro.timeseries.AutoFilterReaderEngine.AutoFilterReader._unfiltered_data` method
- the :py:meth:`~pyaro.timeseries.AutoFilterReaderEngine.AutoFilterReader._unfiltered_stations` method
- the :py:meth:`~pyaro.timeseries.AutoFilterReaderEngine.AutoFilterReader._unfiltered_variables` method
Expand Down
Binary file added docs/pics/Filter.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/pics/Filter.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
49 changes: 41 additions & 8 deletions docs/puml/Filter.puml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ interface timeseries.Filter {
}

class timeseries.VariableNameFilter {
+ __init__(reader_to_new, include, exclude): VariableNameChangingFilter
+ __init__(reader_to_new, include, exclude)
+ name(): "variables"
+ init_kwargs(): dict
+ filter_variables(variables): [str]
Expand All @@ -48,31 +48,31 @@ abstract class timeseries.StationReductionFilter {
}

class timeseries.StationFilter {
+ __init__(include=[station], exclude=[station]): StationFilter
+ __init__(include=[station], exclude=[station])
+ name(): "stations"
+ init_kwargs(): dict
+ filter_stations(stations): dict[str, Station]
has_station(station): bool
}

class timeseries.CountryFilter {
+ __init__(include=[country], exclude=[country]): CountryFilter
+ __init__(include=[country], exclude=[country])
+ name(): "countries"
+ init_kwargs(): dict
+ filter_stations(stations): dict[str, Station]
has_country(iso2_str): bool
}

class timeseries.BoundingBoxFilter {
+ __init__(include=[country], exclude=[country]): BoundingBoxFilter
+ __init__(include=[country], exclude=[country])
+ name(): "bounding_boxes"
+ init_kwargs(): dict
+ filter_stations(stations): dict[str, Station]
has_location(lat, lon): bool
}

class timeseries.TimeBoundsFilter {
+ __init__(start_include=[flag], start_exclude=[flag],\n start_end_include, startend_exclude,\n end_include, end_exclude): TimeBoundsFilter
+ __init__(start_include=[flag], start_exclude=[flag],\n start_end_include, startend_exclude,\n end_include, end_exclude)
+ name(): "time_bounds"
+ init_kwargs(): dict
+ filter_data_idx(data, stations, variables): Data
Expand All @@ -81,37 +81,70 @@ class timeseries.TimeBoundsFilter {
contains(start_dt_array, end_dt_array): bool-array
}

class timeseries.TimeVariableStationFilter {
+ __init__(exclude_from_csvfile="",\nexclude=[(start, end, variable, station)])
+ name(): "time_variable_station"
+ init_kwargs(): dict
+ filter_data_idx(data, stations, variables): Data
}

class timeseries.TimeResolutionFilter {
+ __init__(resolutions=[resolution_string]): TimeResolutionFilter
+ __init__(resolutions=[resolution_string])
+ name(): "time_resolutions"
+ init_kwargs(): dict
+ filter_data_idx(data, stations, variables): Data
}

class timeseries.DuplicateFilter {
+ __init__(duplicate_keys=["stations", "start_times", "end_times"]): DuplicateFilter
+ __init__(duplicate_keys=["stations",\n"start_times", "end_times"])
+ name(): "duplicates"
+ init_kwargs(): dict
+ filter_data_idx(data, stations, variables): Data
}

class timeseries.FlagFilter {
+ __init__(include=[flag], exclude=[flag]): FlagFilter
+ __init__(include=[flag], exclude=[flag])
+ name(): "flags"
+ init_kwargs(): dict
+ filter_data_idx(data, stations, variables): Data
usable_flags(): set(flags)
}

class timeseries.AltitudeFilter {
+ __init__(min_altitude, max_altitude)
+ name(): "altitude"
+ init_kwargs(): dict
+ filter_stations(stations): dict[str, Station]
}

class timeseries.RelativeAltitudeFilter {
+ __init__(topo_file, topo_var, rdiff)
+ name(): "relaltitude"
+ init_kwargs(): dict
+ filter_stations(stations): dict[str, Station]
}

class timeseries.ValleyFloorRelativeAltitudeFilter {
+ __init__(topo_file, radius,topo_var, lower, upper)
+ name(): "valleyfloor_relaltitude"
+ init_kwargs(): dict
+ filter_stations(stations): dict[str, Station]
}



timeseries.Filter <|.. timeseries.VariableNameFilter
timeseries.Filter <|.. timeseries.DataIndexFilter
timeseries.DataIndexFilter <|.. timeseries.StationReductionFilter
timeseries.StationReductionFilter <|.. timeseries.StationFilter
timeseries.StationReductionFilter <|.. timeseries.CountryFilter
timeseries.StationReductionFilter <|.. timeseries.BoundingBoxFilter
timeseries.StationReductionFilter <|.. timeseries.AltitudeFilter
timeseries.StationReductionFilter <|.. timeseries.RelativeAltitudeFilter
timeseries.StationReductionFilter <|.. timeseries.ValleyFloorRelativeAltitudeFilter
timeseries.DataIndexFilter <|.. timeseries.TimeBoundsFilter
timeseries.DataIndexFilter <|.. timeseries.TimeResolutionFilter
timeseries.DataIndexFilter <|.. timeseries.TimeVariableStationFilter
timeseries.DataIndexFilter <|.. timeseries.DuplicateFilter
timeseries.DataIndexFilter <|.. timeseries.FlagFilter

Expand Down
2 changes: 1 addition & 1 deletion src/pyaro/plugins.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ def list_timeseries_engines() -> dict[str, TimeseriesEngine]:

Notes
-----
This function lives in the backends namespace (``engs=pyaro.list_timeseries_enginess()``).
This function lives in the backends namespace (``engs=pyaro.list_timeseries_engines()``).
More information about each reader is available via the TimeseriesEngine obj.url() and
obj.description()

Expand Down
44 changes: 25 additions & 19 deletions src/pyaro/timeseries/Filter.py
Original file line number Diff line number Diff line change
Expand Up @@ -551,20 +551,22 @@ class TimeBoundsFilter(DataIndexFilter):

Examples:

end_include: [("2023-01-01 10:00:00", "2024-01-01 07:00:00")]
end_include: `[("2023-01-01 10:00:00", "2024-01-01 07:00:00")]`
will only include observations where the end time of each observation
is within the interval specified
(i.e. "end" >= 2023-01-01 10:00:00 and "end" <= "2024-01-01 07:00:00")

Including multiple bounds will act as an OR, allowing multiple selections.
If we want every observation in January for 2021, 2022, 2023, and 2024 this
could be made as the following filter:
startend_include: [
("2021-01-01 00:00:00", "2021-02-01 00:00:00"),
("2022-01-01 00:00:00", "2022-02-01 00:00:00"),
("2023-01-01 00:00:00", "2023-02-01 00:00:00"),
("2024-01-01 00:00:00", "2024-02-01 00:00:00"),
]
could be made as the following filter::

startend_include: [
("2021-01-01 00:00:00", "2021-02-01 00:00:00"),
("2022-01-01 00:00:00", "2022-02-01 00:00:00"),
("2023-01-01 00:00:00", "2023-02-01 00:00:00"),
("2024-01-01 00:00:00", "2024-02-01 00:00:00"),
]

"""

def __init__(
Expand Down Expand Up @@ -711,9 +713,11 @@ class TimeVariableStationFilter(DataIndexFilter):
:param exclude_from_csvfile: this is a helper option to enable a large list of excludes
to be read from a "\t" separated file with columns
start \t end \t variable \t station

where start and end are timestamps of format YYYY-MM-DD HH:MM:SS in UTC, e.g.
the year 2020 is:
2020-01-01 00:00:00 \t 2020-12-31 23:59:59 \t ...

"""

def __init__(self, exclude=[], exclude_from_csvfile=""):
Expand Down Expand Up @@ -955,9 +959,9 @@ class RelativeAltitudeFilter(StationFilter):
Filter class which filters stations based on the relative difference between
the station altitude, and the gridded topography altitude.

:param topo_file : A .nc file from which to read gridded topography data.
:param topo_var : Name of variable that stores altitude.
:param rdiff : Relative difference (in meters).
:param topo_file: A .nc file from which to read gridded topography data.
:param topo_var: Name of variable that stores altitude.
:param rdiff: Relative difference (in meters).

Note:
-----
Expand Down Expand Up @@ -1014,6 +1018,8 @@ def UNITS_METER(self):
"""internal representation of units, don't use

:return: m-unit in internal representation

:meta private:
"""
if self._UNITS_METER is None:
self._UNITS_METER = Unit("m")
Expand All @@ -1023,11 +1029,11 @@ def UNITS_METER(self):
def topography(self):
"""Internal property, don't use.

:raises ModuleNotFoundError: _description_
:raises ModuleNotFoundError: _description_
:raises FilterException: _description_
:raises FilterException: _description_
:raises ModuleNotFoundError: if cf-units or xarray is not installed
:raises FilterException: if topograpy file is not provided
:return: topography as internal representation

:meta private:
"""
if "cf_units" not in sys.modules:
raise ModuleNotFoundError(
Expand Down Expand Up @@ -1086,8 +1092,8 @@ def _find_lat_lon_variables(self, topo_xr):

These are assigned to self._lat, self._lon, respectively for later use.

:param topo_xr xr.Dataset of topography
:return lat, lon DataArrays
:param topo_xr: xr.Dataset of topography
:return: lat, lon DataArrays
"""
for var_name in self._topography.coords:
unit_str = self._topography[var_name].attrs.get("units", None)
Expand All @@ -1107,8 +1113,8 @@ def _find_lat_lon_variables(self, topo_xr):
def _extract_bounding_box(self, lat, lon):
"""
Extract the bounding box of the grid, sets self._boundary_(north|east|south|west)
:param lat latitude (DataArray)
:param lon longitude (DataArray)
:param lat: latitude (DataArray)
:param lon: longitude (DataArray)
"""
self._boundary_west = float(lon.min())
self._boundary_east = float(lon.max())
Expand Down