reorganise doc

ecmwf-lab · Mar 23, 2024 · 0ef4332 · 0ef4332
1 parent 1637da7
commit 0ef4332
Show file tree

Hide file tree

Showing 27 changed files with 125 additions and 97 deletions.
diff --git a/docs/datasets/build.png → docs/building/build.png b/docs/datasets/build.png → docs/building/build.png
diff --git a/docs/datasets/building.yaml → docs/building/building.yaml b/docs/datasets/building.yaml → docs/building/building.yaml
diff --git a/docs/building/filters.rst b/docs/building/filters.rst
@@ -0,0 +1,25 @@
+.. _dataset-filters:
+
+#########
+ Filters
+#########
+
+Filters are used to modify the data or metadata in a dataset.
+
+********
+ select
+********
+
+.. include:: filters/select.rst
+
+********
+ rename
+********
+
+.. include:: filters/rename.rst
+
+**************
+ rotate_winds
+**************
+
+.. include:: filters/rotate_winds.rst
diff --git a/docs/building/filters/rename.rst b/docs/building/filters/rename.rst
@@ -0,0 +1 @@
+TODO
diff --git a/docs/building/filters/rotate_winds.rst b/docs/building/filters/rotate_winds.rst
@@ -0,0 +1 @@
+TODO
diff --git a/docs/building/filters/select.rst b/docs/building/filters/select.rst
@@ -0,0 +1 @@
+TODO
diff --git a/docs/datasets/building.rst → docs/building/introduction.rst b/docs/datasets/building.rst → docs/building/introduction.rst
@@ -1,8 +1,8 @@
 .. _datasets-building:
 
-###################
- Building datasets
-###################
+##############
+ Introduction
+##############
 
 ..
    .. figure:: build.png

diff --git a/docs/building/sources.rst b/docs/building/sources.rst
@@ -0,0 +1,41 @@
+.. _dataset-sources:
+
+#########
+ Sources
+#########
+
+******
+ mars
+******
+
+.. include:: sources/mars.rst
+
+******
+ grib
+******
+
+.. include:: sources/grib.rst
+
+********
+ netcdf
+********
+
+.. include:: sources/netcdf.rst
+
+*********
+ opendap
+*********
+
+.. include:: sources/opendap.rst
+
+***********
+ constants
+***********
+
+.. include:: sources/constants.rst
+
+***************
+ accumulations
+***************
+
+.. include:: sources/accumulations.rst
diff --git a/docs/building/sources/accumulations.rst b/docs/building/sources/accumulations.rst
@@ -0,0 +1 @@
+TODO
diff --git a/docs/building/sources/constants.rst b/docs/building/sources/constants.rst
@@ -0,0 +1 @@
+TODO
diff --git a/docs/building/sources/grib.rst b/docs/building/sources/grib.rst
@@ -0,0 +1 @@
+TODO
diff --git a/docs/building/sources/mars.rst b/docs/building/sources/mars.rst
@@ -0,0 +1 @@
+TODO
diff --git a/docs/building/sources/netcdf.rst b/docs/building/sources/netcdf.rst
@@ -0,0 +1 @@
+TODO
diff --git a/docs/building/sources/opendap.rst b/docs/building/sources/opendap.rst
@@ -0,0 +1 @@
+TODO
diff --git a/docs/datasets/about.rst b/docs/datasets/about.rst
diff --git a/docs/datasets/filters.rst b/docs/datasets/filters.rst
diff --git a/docs/datasets/sources.rst b/docs/datasets/sources.rst
diff --git a/docs/datasets/using.rst b/docs/datasets/using.rst
diff --git a/docs/datasets/images.pptx → docs/images.pptx b/docs/datasets/images.pptx → docs/images.pptx
diff --git a/docs/index.rst b/docs/index.rst
@@ -31,26 +31,33 @@ models from existing recipes but with their own data.
    firststeps
    examples
 
-**Datasets**
+**Using training datasets**
 
--  :doc:`datasets/about`
--  :doc:`datasets/building`
--  :doc:`datasets/sources`
--  :doc:`datasets/filters`
--  :doc:`datasets/using`
--  :doc:`datasets/options`
+-  :doc:`using/introduction`
+-  :doc:`using/options`
 
 .. toctree::
    :maxdepth: 1
    :hidden:
-   :caption: Training datasets
+   :caption: Using datasets
 
-   datasets/about
-   datasets/building
-   datasets/sources
-   datasets/filters
-   datasets/using
-   datasets/options
+   using/introduction
+   using/options
+
+**Building training datasets**
+
+-  :doc:`building/introduction`
+-  :doc:`building/sources`
+-  :doc:`building/filters`
+
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+   :caption: Building datasets
+
+   building/introduction
+   building/sources
+   building/filters
 
 *********
  License
@@ -62,6 +69,4 @@ models from existing recipes but with their own data.
 
 .. _ecml-tools: https://github.com/ecmwf-lab/ecml-tools
 
-.. _pytorch: https://pytorch.org
-
 .. _zarr: https://zarr.readthedocs.io/
diff --git a/docs/datasets/data.png → docs/overview.png b/docs/datasets/data.png → docs/overview.png
diff --git a/docs/overview.rst b/docs/overview.rst
@@ -2,5 +2,26 @@
  Overview
 ##########
 
-This documentation is currently being written. For now, only the
-creation and use of training datasets is documented.
+Training datasets are large array-like objects encode in Zarr_ format.
+They
+
+The array has the following dimensions:
+
+.. figure:: overview.png
+   :alt: Data layout
+
+The first dimension is the time dimension, the second dimension are the
+variables (e.g. temperature, pressure, etc), the third dimension is the
+ensemble, and fourth dimension are the grid points values.
+
+This structure provides an efficient way to build the training dataset,
+as input and output of the model are simply consecutive slices of the
+array.
+
+.. code:: python
+
+   x, y  = ds[n], ds[n+1]
+   y_hat = model.predict(x)
+   loss  = model.loss(y, y_hat)
+
+.. _zarr: https://zarr.readthedocs.io/
diff --git a/docs/datasets/concat.png → docs/using/concat.png b/docs/datasets/concat.png → docs/using/concat.png
diff --git a/docs/using/introduction.rst b/docs/using/introduction.rst
@@ -0,0 +1,3 @@
+##############
+ Introduction
+##############
diff --git a/docs/datasets/join.png → docs/using/join.png b/docs/datasets/join.png → docs/using/join.png
diff --git a/docs/datasets/options.rst → docs/using/options.rst b/docs/datasets/options.rst → docs/using/options.rst
diff --git a/docs/datasets/overlay.png → docs/using/overlay.png b/docs/datasets/overlay.png → docs/using/overlay.png