Schemaupfront (#154)

* add test streams view for BRAT * must cascade drop of lookup when reloading to drop dependent view * add discharge scripts * move lookups * add precip scripts * add channel width scripts * download full bcfishpass file for now * clean up extras scripts * fix paths, minor errors * use st_intersect instead of buffer/coveredby for much better speed * fix psql call * zip lookups * load discharge * fix #150 * by removing wsg, streams are no longer missed * note rasterstats requirement, fix temp table names * update cw measured calculation * complete cw move * update streams view * create extras tables in setup ddl (truncate on load), dump extras csvs * reorganize to separate table creation from load * more cleanup * remove table creation * fix #151 * document streams vw, simpify setup doc to reduce redundancy * remove docker info from primary readme * add missing function to build, update quickstart examples * tables now holds views as well, bump version * remove pgfs grants * add downstream trace, pull trace data from source tables not view, add upstr trace test, make column order in traces match column order in streams_vw * fix dnstr trace test * fix dockerfile
smnorris · Feb 18, 2024 · 69d1c4e · 69d1c4e
1 parent 5ff4b2c
commit 69d1c4e
Show file tree

Hide file tree

Showing 103 changed files with 3,589 additions and 1,573 deletions.
diff --git a/.gitignore b/.gitignore
@@ -9,6 +9,11 @@
 # data files
 data/
 *.zip
+*.gz
+extras/channel_width/*csv
+extras/discharge/*csv
+extras/lookups/*csv
+extras/precipitation/*csv
 docs/hydro*.csv
 
 # docker postgres volume

diff --git a/CHANGES.txt b/CHANGES.txt
@@ -3,6 +3,13 @@ Changes
 
 All issue numbers are relative to https://github.com/smnorris/fwapg/issues
 
+0.6.0 ()
+------------------
+- add modelled discharge / channel width / precipitation scripts and data
+- add streams_vw (with mean annual discharge / channel width / upstream mean annual precip)
+- move parent order / max order columns from primary streams table to streams_vw
+- consolidated database setup sql for easier maintenance
+
 0.5.1 (2023-06-22)
 ------------------
 - add missing files for FWA_UpstreamTrace

diff --git a/Dockerfile b/Dockerfile
@@ -17,9 +17,10 @@ RUN apt-get update && apt-get --assume-yes upgrade \
     && apt-get -qq install -y --no-install-recommends python3-psycopg2 \
     && pip3 install --upgrade numpy \
     && pip3 install bcdata \
+    && pip3 install rasterstats \
     && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /home/fwapg
-COPY ["sql", "sql/"]
+COPY ["db", "db/"]
 COPY ["extras", "extras/"]
 COPY [".env.docker", "Makefile", "./"]
diff --git a/Makefile b/Makefile
diff --git a/README.md b/README.md
@@ -10,7 +10,7 @@
 - provide `gradient` values for every FWA stream
 - enable quickly serving FWA features as vector tiles (MVT)
 - enable quickly serving FWA features and custom fwapg functions
-- using [`bcdata`](https://github.com/smnorris/bcdata), extract FWA data from DataBC WFS for easy data updates
+- link additional data to FWA streams (PCIC mean annual discharge, modelled channel width, upstream precipitation)
 
 See [documentation](https://smnorris.github.io/fwapg/) for setup and usage details, plus table and function references.
 
@@ -34,58 +34,39 @@ See [documentation](https://smnorris.github.io/fwapg/) for setup and usage detai
         cd fwapg
         make
 
-The full load takes about 2 hours - but once complete, you can run `fwapg` enabled queries with your favorite sql client. For example:
+The full load takes some time - but once complete, you can run `fwapg` enabled queries with your favorite sql client. For example:
 
 *Locate the nearest point on the FWA stream network to a X,Y location on Highway 14:*
 
-        SELECT gnis_name, blue_line_key, downstream_route_measure
-        FROM FWA_IndexPoint(-123.7028, 48.3858, 4326);
+        SELECT 
+          gnis_name, 
+          blue_line_key, 
+          downstream_route_measure
+        FROM FWA_IndexPoint(ST_Transform(ST_GeomFromText('POINT(-123.7028 48.3858)', 4326), 3005));
 
           gnis_name  | blue_line_key | downstream_route_measure
         -------------+---------------+--------------------------
          Sooke River |     354153927 |        350.2530543284006
 
-*Generate the watershed upstream of above location:*
+*Generate the watershed upstream of this location:*
 
         SELECT ST_ASText(geom) FROM FWA_WatershedAtMeasure(354153927, 350);
 
          st_astext
         --------------
         POLYGON((...
 
+*Select all stream upstream of this location:*
 
-See [Usage](https://smnorris.github.io/fwapg/02_usage.html) for more examples.
-
-
-## Docker
-
-Download the repo, create containers, create database, load fwa data:
-
-    git clone https://github.com/smnorris/fwapg.git
-    cd fwapg
-    docker-compose build
-    docker-compose up -d
-    docker-compose run --rm loader psql -c "CREATE DATABASE fwapg" postgres
-    docker-compose run --rm loader make --debug=basic
-
-Note that docker images specified in `docker-compose.yml` may not be available on ARM based systems.
-As long as you do not remove the container `fwapg-db`, it will retain all the data you put in it.
-If you have shut down Docker or the container, start it up again with this command:
-
-    docker-compose up -d
+        SELECT ST_ASText(geom)
+        FROM FWA_UpstreamTrace(354153927, 350);
 
-Connect to the db from your host OS via the port specified in `docker-compose.yml`:
-
-    psql -p 8000 -U postgres fwapg
-
-Or see the FWA data in the browser as vector tiles/geojson features:
-
-    http://localhost:7800/
-    http://localhost:9000/
+         st_astext
+        --------------
+        LINESTRINGZM((...
 
-Delete the containers (and associated fwa data):
 
-    docker-compose down
+See [Usage](https://smnorris.github.io/fwapg/02_usage.html) for more examples.
 
 
 ## Tile and feature services

diff --git a/sql/misc/drop_all.sql → db/clean.sql b/sql/misc/drop_all.sql → db/clean.sql
@@ -36,6 +36,12 @@ drop table if exists whse_basemapping.fwa_streams_watersheds_lut;
 drop table if exists whse_basemapping.fwa_waterbodies_upstream_area;
 drop table if exists whse_basemapping.fwa_watersheds_upstream_area;
 
+drop table if exists whse_basemapping.fwa_stream_networks_order_max;
+drop table if exists whse_basemapping.fwa_stream_networks_order_parent;
+drop table if exists whse_basemapping.fwa_stream_networks_discharge;
+drop table if exists whse_basemapping.fwa_stream_networks_mean_annual_precip;
+drop table if exists whse_basemapping.fwa_stream_networks_channel_width;
+
 -- additional data
 drop table if exists usgs.wbdhu12;
 drop table if exists hydrosheds.hybas_lev12_v1c;
@@ -61,5 +67,4 @@ drop function if exists usgs.huc12;
 drop function if exists hydrosheds.hydroshed;
 drop function if exists st_safe_repair;
 
-drop schema if exists postgisftw cascade;
 drop schema if exists fwapg cascade;
diff --git a/db/extensions.sql b/db/extensions.sql
@@ -0,0 +1,3 @@
+create extension if not exists postgis with schema public;
+create extension if not exists ltree with schema public;
+create extension if not exists intarray with schema public;
diff --git a/sql/functions/CDB_MakeHexagon.sql → db/functions/CDB_MakeHexagon.sql b/sql/functions/CDB_MakeHexagon.sql → db/functions/CDB_MakeHexagon.sql
diff --git a/sql/functions/FWA_Downstream.sql → db/functions/FWA_Downstream.sql b/sql/functions/FWA_Downstream.sql → db/functions/FWA_Downstream.sql
diff --git a/db/functions/FWA_DownstreamTrace.sql b/db/functions/FWA_DownstreamTrace.sql
@@ -0,0 +1,192 @@
+--DROP FUNCTION fwa_downstreamtrace(integer,double precision,double precision);
+-- -------------------------------------------------------------------------------------------------------------------------
+-- FWA_DownstreamTrace
+-- Return complete stream network downstream of provided location
+-- (breaking stream at given location if location is farther from existing endpoint than the provided tolerance)
+-- NOTE - features with null local codes are not returned -
+-- -------------------------------------------------------------------------------------------------------------------------
+CREATE OR REPLACE FUNCTION whse_basemapping.FWA_DownstreamTrace(
+  start_blue_line_key integer,
+  start_measure float,
+  tolerance float default 1
+)
+
+RETURNS TABLE (
+  linear_feature_id        bigint                      ,
+  edge_type                integer                     ,
+  blue_line_key            integer                     ,
+  watershed_key            integer                     ,
+  wscode                   ltree                       ,
+  localcode                ltree                       ,
+  watershed_group_code     character varying(4)        ,
+  downstream_route_measure double precision            ,
+  upstream_route_measure   double precision            ,
+  length_metre             double precision            ,
+  waterbody_key            integer                     ,
+  gnis_name                character varying(80)       ,
+  stream_order             integer                     ,
+  stream_magnitude         integer                     ,
+  feature_code             character varying(10)       ,
+  gradient                 double precision            ,
+  left_right_tributary     character varying(7)        ,
+  stream_order_parent      integer                     ,
+  stream_order_max         integer                     ,
+  upstream_area_ha         double precision            ,
+  map_upstream             integer                     ,
+  channel_width            double precision            ,
+  channel_width_source     text                        ,
+  mad_m3s                  double precision            ,
+  geom                     geometry(LineStringZM,3005)
+)
+
+AS
+
+$$
+
+DECLARE
+   v_blue_line_key  integer := start_blue_line_key;
+   v_measure        float   := start_measure;
+   v_tolerance      float   := tolerance;
+
+BEGIN
+
+RETURN QUERY
+
+-- find segment on which point lies
+with segment as (
+  select
+    s.linear_feature_id,
+    s.blue_line_key,
+    v_measure AS measure,
+    s.downstream_route_measure,
+    s.upstream_route_measure,
+    s.wscode_ltree as wscode,
+    s.localcode_ltree as localcode,
+    s.geom
+  FROM whse_basemapping.fwa_stream_networks_sp AS s
+  WHERE s.blue_line_key = v_blue_line_key
+  AND round(s.downstream_route_measure::numeric, 4) <= round(v_measure::numeric, 4)
+  AND round(s.upstream_route_measure::numeric, 4) > round(v_measure::numeric, 4)
+),
+
+-- cut segment if required
+cut as (
+  SELECT
+    s.linear_feature_id,
+    s.blue_line_key,
+    case
+      when (s.upstream_route_measure - v_measure) > v_tolerance then s.measure
+      else s.upstream_route_measure
+    end as upstream_route_measure,
+    s.wscode,
+    s.localcode,
+    (st_dump(
+      case
+        when (s.upstream_route_measure - v_measure) > v_tolerance  -- split geom if not within tolerance m of upstr measure
+        then ST_LocateBetween(s.geom, s.downstream_route_measure, v_measure)
+        else s.geom                                                  -- otherwise return source geom
+      end
+    )).geom AS geom
+  FROM segment s
+  WHERE (v_measure - s.downstream_route_measure) > v_tolerance -- only return data if more than tolerance m from dnstr measure
+),
+
+-- find everything downstream
+dnstr as (
+  select a.*
+  from whse_basemapping.fwa_stream_networks_sp a
+  inner join segment b on fwa_downstream(
+    b.blue_line_key,
+    b.downstream_route_measure,
+    b.wscode,
+    b.localcode,
+    a.blue_line_key,
+    a.downstream_route_measure,
+    a.wscode_ltree,
+    a.localcode_ltree
+    )
+)
+
+select
+  s.linear_feature_id,
+  s.edge_type,
+  s.blue_line_key,
+  s.watershed_key,
+  s.wscode_ltree as wscode,
+  s.localcode_ltree as localcode,
+  s.watershed_group_code,
+  s.downstream_route_measure,
+  c.upstream_route_measure,
+  round(st_length(c.geom)::numeric, 7) as length_metre,
+  s.waterbody_key,
+  s.gnis_name,
+  s.stream_order,
+  s.stream_magnitude,
+  s.feature_code,
+  (round(((st_z(st_pointn(c.geom, '-1'::integer)) - st_z(st_pointn(c.geom, 1))) / st_length(c.geom))::numeric, 4)) as gradient,
+  s.left_right_tributary,
+  op.stream_order_parent,
+  om.stream_order_max,
+  ua.upstream_area_ha,
+  p.map_upstream,
+  cw.channel_width,
+  cw.channel_width_source,
+  d.mad_m3s,
+  c.geom as geom
+from cut c
+inner join whse_basemapping.fwa_stream_networks_sp s on c.linear_feature_id = s.linear_feature_id
+left outer join whse_basemapping.fwa_streams_watersheds_lut l on s.linear_feature_id = l.linear_feature_id
+inner join whse_basemapping.fwa_watersheds_upstream_area ua on l.watershed_feature_id = ua.watershed_feature_id
+left outer join whse_basemapping.fwa_stream_networks_channel_width cw on c.linear_feature_id = cw.linear_feature_id
+left outer join whse_basemapping.fwa_stream_networks_discharge d on c.linear_feature_id = d.linear_feature_id
+left outer join whse_basemapping.fwa_stream_networks_mean_annual_precip p ON s.wscode_ltree = p.wscode_ltree AND s.localcode_ltree = p.localcode_ltree
+left outer join whse_basemapping.fwa_stream_networks_order_max om on s.blue_line_key = om.blue_line_key
+left outer join whse_basemapping.fwa_stream_networks_order_parent op on s.blue_line_key = op.blue_line_key
+union all
+select
+  dn.linear_feature_id,
+  s.edge_type,
+  s.blue_line_key,
+  s.watershed_key,
+  s.wscode_ltree as wscode,
+  s.localcode_ltree as localcode,
+  s.watershed_group_code,
+  s.downstream_route_measure,
+  s.upstream_route_measure,
+  s.length_metre,
+  s.waterbody_key,
+  s.gnis_name,
+  s.stream_order,
+  s.stream_magnitude,
+  s.feature_code,
+  s.gradient,
+  s.left_right_tributary,
+  op.stream_order_parent,
+  om.stream_order_max,
+  ua.upstream_area_ha,
+  p.map_upstream,
+  cw.channel_width,
+  cw.channel_width_source,
+  d.mad_m3s,
+  s.geom as geom
+from dnstr dn
+inner join whse_basemapping.fwa_stream_networks_sp s on dn.linear_feature_id = s.linear_feature_id
+left outer join whse_basemapping.fwa_streams_watersheds_lut l on s.linear_feature_id = l.linear_feature_id
+inner join whse_basemapping.fwa_watersheds_upstream_area ua on l.watershed_feature_id = ua.watershed_feature_id
+left outer join whse_basemapping.fwa_stream_networks_channel_width cw on dn.linear_feature_id = cw.linear_feature_id
+left outer join whse_basemapping.fwa_stream_networks_discharge d on dn.linear_feature_id = d.linear_feature_id
+left outer join whse_basemapping.fwa_stream_networks_mean_annual_precip p ON s.wscode_ltree = p.wscode_ltree AND s.localcode_ltree = p.localcode_ltree
+left outer join whse_basemapping.fwa_stream_networks_order_max om on s.blue_line_key = om.blue_line_key
+left outer join whse_basemapping.fwa_stream_networks_order_parent op on s.blue_line_key = op.blue_line_key
+order by wscode desc, localcode desc, downstream_route_measure desc;
+
+
+END
+
+$$
+LANGUAGE 'plpgsql' IMMUTABLE STRICT PARALLEL SAFE;
+
+COMMENT ON FUNCTION whse_basemapping.FWA_DownstreamTrace IS 'Return stream network downstream of provided location';
+
+
+
diff --git a/sql/functions/FWA_IndexPoint.sql → db/functions/FWA_IndexPoint.sql b/sql/functions/FWA_IndexPoint.sql → db/functions/FWA_IndexPoint.sql
diff --git a/sql/functions/FWA_LocateAlong.sql → db/functions/FWA_LocateAlong.sql b/sql/functions/FWA_LocateAlong.sql → db/functions/FWA_LocateAlong.sql
diff --git a/sql/functions/FWA_LocateAlongInterval.sql → db/functions/FWA_LocateAlongInterval.sql b/sql/functions/FWA_LocateAlongInterval.sql → db/functions/FWA_LocateAlongInterval.sql
diff --git a/sql/functions/FWA_SliceWatershedAtPoint.sql → db/functions/FWA_SliceWatershedAtPoint.sql b/sql/functions/FWA_SliceWatershedAtPoint.sql → db/functions/FWA_SliceWatershedAtPoint.sql
diff --git a/sql/functions/FWA_StreamsAsMVT.sql → db/functions/FWA_StreamsAsMVT.sql b/sql/functions/FWA_StreamsAsMVT.sql → db/functions/FWA_StreamsAsMVT.sql
diff --git a/sql/functions/FWA_Upstream.sql → db/functions/FWA_Upstream.sql b/sql/functions/FWA_Upstream.sql → db/functions/FWA_Upstream.sql
diff --git a/...functions/FWA_UpstreamBorderCrossings.sql → db/functions/FWA_UpstreamBorderCrossings.sql b/...functions/FWA_UpstreamBorderCrossings.sql → db/functions/FWA_UpstreamBorderCrossings.sql