0.2 (#2)

* edit dbt_project.yml * move model to integration_tests folder * add integration test workflow * update integration_test dbt_project.yml * update gitignore and README * POC on error handler * error macros working, fixed required_docs to respect false config * very heavy WIP * WIP: dispatch, marco prfxs, data tests working * refactored errors to return and handle in caller * touch ci * touch ci * add profiles and envs * tests_per_model edits - broken * Revert "add profiles and envs" This reverts commit 605d5de. * Revert "tests_per_model edits - broken" This reverts commit 4febca9. * passing cases * touch README * added start msg * updated README and version * ci fix * ci
tnightengale · Mar 1, 2021 · a3f60f9 · a3f60f9
1 parent cccdd92
commit a3f60f9
Show file tree

Hide file tree

Showing 41 changed files with 800 additions and 229 deletions.
diff --git a/.github/workflows/integration_tests.yml b/.github/workflows/integration_tests.yml
@@ -0,0 +1,36 @@
+name: Integration Tests
+on:
+  # Triggers the workflow on push or pull request events but only for the master branch
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  ci:
+    runs-on: ubuntu-latest
+    env:
+      DBT_PROFILES_DIR: . # Use integration_tests/profiles.yml
+
+    steps:
+    - name: Checkout Branch
+      uses: actions/checkout@v2
+
+    - name: Install Python
+      uses: actions/setup-python@v2
+      with: 
+        python-version: 3.8
+
+    - name: Install dbt 0.19.0
+      run: pip install dbt==0.19.0
+
+    - name: Run and Test
+      env: 
+        CI_SNOWFLAKE_DBT_ACCOUNT: ${{ secrets.CI_SNOWFLAKE_DBT_ACCOUNT }}
+        CI_SNOWFLAKE_DBT_USER: ${{ secrets.CI_SNOWFLAKE_DBT_USER }}
+        CI_SNOWFLAKE_DBT_PASS: ${{ secrets.CI_SNOWFLAKE_DBT_PASS }}
+        CI_SNOWFLAKE_DBT_ROLE: ${{ secrets.CI_SNOWFLAKE_DBT_ROLE }}
+        CI_SNOWFLAKE_DBT_DATABASE: ${{ secrets.CI_SNOWFLAKE_DBT_DATABASE }}
+        CI_SNOWFLAKE_DBT_WAREHOUSE: ${{ secrets.CI_SNOWFLAKE_DBT_WAREHOUSE }}
+      run: |
+        cd integration_tests
+        dbt deps
+        dbt test --data
diff --git a/.gitignore b/.gitignore
@@ -4,4 +4,4 @@ dbt_modules/
 logs/
 .env
 .user.yml
-profiles.yml
+.vscode
diff --git a/README.md b/README.md
@@ -1,21 +1,42 @@
+https://github.com/tnightengale/dbt-meta-testing/workflows/Integration%20Tests/badge.svg
+
 # dbt Meta Testing
-This dbt package contains macros to assert test and documentation coverage from `dbt_project.yml` configuration settings.
+This dbt package contains macros to assert test and documentation coverage from
+`dbt_project.yml` configuration settings.
+
+## Table of Contents
+  - [Install](#install)
+  - [Configurations](#configurations)
+    - [**Required Tests**](#required-tests)
+    - [**Required Docs**](#required-docs)
+  - [Usage](#usage)
+    - [required_tests (source)](#required_tests-source)
+    - [required_docs (source)](#required_docs-source)
+    - [dbt_meta_testing.logger (source)](#dbt_meta_testinglogger-source)
+  - [Contributions](#contributions)
+  - [Testing](#testing)
+    - [Verified Data Warehouses](#verified-data-warehouses)
 
 ## Install
 Include in `packages.yml`:
 
 ```yaml
 packages:
-  - git: "https://github.com/tnightengale/quality-assurance-dbt"
-    revision: 0.1.2
+  - package: tnightengale/dbt_meta_testing
+    version: 0.2.0
 ```
-For latest release, see https://github.com/tnightengale/dbt-meta-testing/releases.
+For latest release, see
+https://github.com/tnightengale/dbt-meta-testing/releases.
 
 ## Configurations
-This package features two meta configs that can be applied to a dbt project: `+required_tests` and `+required_docs`. Read the dbt documentation [here](https://docs.getdbt.com/reference/model-configs) to learn more about model configurations in dbt.
+This package features two meta configs that can be applied to a dbt project:
+`+required_tests` and `+required_docs`. Read the dbt documentation
+[here](https://docs.getdbt.com/reference/model-configs) to learn more about
+model configurations in dbt.
 
 ### **Required Tests**
-To require test coverage, define the `+required_tests` configuration on a model path in `dbt_project.yml`:
+To require test coverage, define the `+required_tests` configuration on a model
+path in `dbt_project.yml`:
 ```yaml
 # dbt_project.yml
 ...
@@ -27,7 +48,9 @@ models:
             +required_tests: {"unique": 1}
 ```
 
-The `+required_tests` config must be either a `dict` or `None`. All the regular dbt configuration hierarchy rules apply. For example, individual model configs will override configs from the `dbt_project.yml`:
+The `+required_tests` config must be either a `dict` or `None`. All the regular
+dbt configuration hierarchy rules apply. For example, individual model configs
+will override configs from the `dbt_project.yml`:
 ```sql
 -- /models/marts/core/your_model.sql
 {{
@@ -37,19 +60,26 @@ The `+required_tests` config must be either a `dict` or `None`. All the regular
 SELECT
 ...
 ```
-The provided dictionary can contain any column schema test as a key, followed by the minimum number of occurances which must be included on the model. In the example above, every model in the `models/marts/` path must include at least one `unique` test.
-
-Custom column-level schema tests are supported. However, in order to appear in the `graph` context variable (which this package parses), they must be applied to at least one model in the project prior to compilation. 
-
-Model-level schema tests are currently _not supported_. For example the following model-level `dbt_utils.equal_rowcount` test _cannot_ currently be asserted via the configuration:
+The provided dictionary can contain any column schema test as a key, followed by
+the minimum number of occurances which must be included on the model. In the
+example above, every model in the `models/marts/` path must include at least one
+`unique` test.
+
+Custom column-level schema tests are supported. However, in order to appear in
+the `graph` context variable (which this package parses), they must be applied
+to at least one model in the project prior to compilation. 
+
+Model-level schema tests are currently _not supported_. For example the
+following model-level `dbt_utils.equal_rowcount` test _cannot_ currently be
+asserted via the configuration:
 ```yaml   
 # models/schema.yml
 ...
-    - name: my_second_dbt_model
+    - name: model_2
       description: ""
       tests:
         - dbt_utils.equal_rowcount:
-            compare_model: ref('my_first_dbt_model')
+            compare_model: ref('model_1')
       columns:
           - name: id
             description: "The primary key for this table"
@@ -59,14 +89,15 @@ Model-level schema tests are currently _not supported_. For example the followin
                 - mock_schema_test
 ```
 
-Models that do not meet their configured test minimums will be listed in the error when validated via a `run-operation`:
+Models that do not meet their configured test minimums will be listed in the
+error when validated via a `run-operation`:
 ```
 usr@home dbt-meta-testing $ dbt run-operation required_tests
 Running with dbt=0.18.1
 Encountered an error while running operation: Compilation Error in macro required_tests (macros/required_tests.sql)
   Insufficient test coverage from the 'required_tests' config on the following models: 
-  Model: 'my_first_dbt_model' Test: 'not_null' Got: 1 Expected: 2
-  Model: 'my_first_dbt_model' Test: 'mock_schema_test' Got: 0 Expected: 1
+  Model: 'model_1' Test: 'not_null' Got: 1 Expected: 2
+  Model: 'model_1' Test: 'mock_schema_test' Got: 0 Expected: 1
   
   > in macro _evaluate_required_tests (macros/utils/required_tests/evaluate_required_tests.sql)
   > called by macro required_tests (macros/required_tests.sql)
@@ -75,7 +106,8 @@ usr@home dbt-meta-testing $
 ```
 
 ### **Required Docs**
-To require documentation coverage, define the `+required_docs` configuration on a model path in `dbt_project.yml`:
+To require documentation coverage, define the `+required_docs` configuration on
+a model path in `dbt_project.yml`:
 ```yaml
 # dbt_project.yml
 ...
@@ -96,7 +128,7 @@ For example, the following configurations:
 version: 2
 
 models:
-    - name: my_first_dbt_model
+    - name: model_1
       description: "A starter dbt model"
       columns:
           - name: id
@@ -105,11 +137,11 @@ models:
                 - unique
                 - not_null
 
-    - name: my_second_dbt_model
+    - name: model_2
       description: ""
       tests:
         - dbt_utils.equal_rowcount:
-            compare_model: ref('my_first_dbt_model')
+            compare_model: ref('model_1')
       columns:
           - name: id
             description: "The primary key for this table"
@@ -119,13 +151,13 @@ models:
 
 ```
 
-Where `my_second_dbt_model` has a column `new` which is not defined in the `.yml` above:
+Where `model_2` has a column `new` which is not defined in the `.yml` above:
 ```sql
--- models/example/my_second_dbt_model.sql
+-- models/example/model_2.sql
 select 
     *,
     'new' as new
-from {{ ref('my_first_dbt_model') }}
+from {{ ref('model_1') }}
 where id = 1
 ```
 
@@ -145,11 +177,11 @@ usr@home dbt-meta-testing $ dbt run-operation required_docs
 Running with dbt=0.18.1
 Encountered an error while running operation: Compilation Error in macro required_docs (macros/required_docs.sql)
   The following models are missing descriptions:
-   - my_second_dbt_model
+   - model_2
   The following columns are missing from the model yml:
-   - my_second_dbt_model.new
+   - model_2.new
   The following columns are present in the model yml, but have blank descriptions:
-   - my_first_dbt_model.id
+   - model_1.id
   
   > in macro _evaluate_required_docs (macros/utils/required_docs/evaluate_required_docs.sql)
   > called by macro required_docs (macros/required_docs.sql)
@@ -158,47 +190,63 @@ usr@home dbt-meta-testing $
 ```
 
 ## Usage
-To assert either the `+required_tests` or `+required_docs` configuration, run the correpsonding macro as a `run-operation` within the dbt CLI.
+To assert either the `+required_tests` or `+required_docs` configuration, run
+the correpsonding macro as a `run-operation` within the dbt CLI.
 
-By default the macro will check all models with the corresponding configuration. If any model does not meet the configuration, the `run-operation` will fail (non-zero) and display an appropriate error message.
+By default the macro will check all models with the corresponding configuration.
+If any model does not meet the configuration, the `run-operation` will fail
+(non-zero) and display an appropriate error message.
 
-To assert the configuration for only a subset of the configured models (eg. new models only in a CI) define the dbt var `models` as a space delimited string of model names to use. 
+To assert the configuration for only a subset of the configured models (eg. new
+models only in a CI) pass an argument, `models`, to the macro as a space
+delimited string of model names to use. 
 
-It's also possible to pass in the result of a `dbt ls -m <selection_syntax>` command, in order to make use of [dbt node selection syntax](https://docs.getdbt.com/reference/node-selection/syntax). Use shell subsitution in a dictionary representation.
+It's also possible to pass in the result of a `dbt ls -m <selection_syntax>`
+command, in order to make use of [dbt node selection
+syntax](https://docs.getdbt.com/reference/node-selection/syntax). Use shell
+subsitution in a dictionary representation.
 
 For example, to run only changed models using dbt's Slim CI feature:
 ```bash
-$ dbt run-operation required_tests --vars "{'models':'$(dbt list -m state:modified --state <filepath>)'}"
+$ dbt run-operation required_tests --args "{'models':'$(dbt list -m state:modified --state <filepath>)'}"
 ```
 
-Alternatively, it's possible to use `git diff` to find changed models; a space delimited string of model names will work as well:
+Alternatively, it's possible to use `git diff` to find changed models; a space
+delimited string of model names will work as well:
 ```bash
-$ dbt run-operation required_tests --vars "{'models':'model1 model2 model3'}"
+$ dbt run-operation required_tests --args "{'models':'model1 model2 model3'}"
 ```
 
 ### required_tests ([source](macros/required_tests.sql))
-Validates that models meet the `+required_tests` configurations applied in `dbt_project.yml`. Typically used only as a `run-operation` in a CI pipeline.
+Validates that models meet the `+required_tests` configurations applied in
+`dbt_project.yml`. Typically used only as a `run-operation` in a CI pipeline.
 
 Usage:
 ```
-$ dbt run-operation required_tests [--vars "{'models': '<space_delimited_models>'}"]
+$ dbt run-operation required_tests [--args "{'models': '<space_delimited_models>'}"]
 ```
 
 ### required_docs ([source](macros/required_tests.sql))
-Validates that models meet the `+required_docs` configurations applied in `dbt_project.yml`. Typically used only as a `run-operation` in a CI pipeline.
+Validates that models meet the `+required_docs` configurations applied in
+`dbt_project.yml`. Typically used only as a `run-operation` in a CI pipeline.
 
 
 Usage:
 ```
-$ dbt run-operation required_docs [--vars "{'models': '<space_delimited_models>'}"]
+$ dbt run-operation required_docs [--args "{'models': '<space_delimited_models>'}"]
 ```
-**Note:** Run this command _after_ `dbt run`: only models that already exist in the warehouse can be validated for columns that are missing from the model `.yml`.
+**Note:** Run this command _after_ `dbt run`: only models that already exist in
+the warehouse can be validated for columns that are missing from the model
+`.yml`.
 
-### logger ([source](macros/logger.sql))
-An ammenity macro that mimics pythons logging module. The level is passed via the `log_level` kwarg. The default "method" (ie. `log_level`) is `DEBUG`, similar to calling `logging.debug('some buggy thing')` in python.
+### dbt_meta_testing.logger ([source](macros/logger.sql))
+An ammenity macro that mimics pythons logging module. The level is passed via
+the `log_level` kwarg. The default "method" (ie. `log_level`) is `DEBUG`,
+similar to calling `logging.debug('some buggy thing')` in python.
 
 
-Set a level with a project var, similar to `logging.basicConfig(level=logging.INFO)` in python:
+Set a level with a project var, similar to
+`logging.basicConfig(level=logging.INFO)` in python:
 ```yaml
 # dbt_project.yml
 ...
@@ -216,17 +264,34 @@ Usage:
 {% if my_object is mapping %}
 
      -- This will be a DEBUG call by default, it won't show up if logging_level="INFO" or higher
-    {{ logger("my_object is mapping: " ~ my_object is mapping) }}
+    {{ dbt_meta_testing.logger("my_object is mapping: " ~ my_object is mapping) }}
 
     ...
 
 {% endif %}
 
 -- This will be an INFO call, it will show up if logging_level="INFO" or lower
-{{ logger("The following keys were found: " ~ my_complex_object.keys(), log_level="INFO") }}
+{{ dbt_meta_testing.logger("The following keys were found: " ~ my_complex_object.keys(), log_level="INFO") }}
 
 {{ return(my_complex_object) }}
 ```
 
 ## Contributions
-Feedback on this project is welcomed and encouraged. Please open an issue or start a discussion if you would like to request a feature change or contribute to this project. 
+Feedback on this project is welcomed and encouraged. Please open an issue or
+start a discussion if you would like to request a feature change or contribute
+to this project. 
+
+## Testing
+The integration tests for this package are located at
+[./integration_tests/tests/](integration_tests/tests/).
+
+To run the tests locally, ensure you have the correct environment variables set
+according to the targets in
+[./integration_tests/profiles.yml](integration_tests/profiles.yml) and use:
+```bash
+$ dbt test --data
+```
+
+### Verified Data Warehouses
+This package has been tested for the following data warehouses:
+- Snowflake
diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,15 +1,15 @@
 
 # Project name.
 name: 'dbt_meta_testing'
-version: '0.1.2'
+version: '0.2.0'
 config-version: 2
+require-dbt-version: ">=0.19.0"
 
 # The "profile" dbt uses for this project.
 profile: 'dbt_meta_testing'
 
 # Configuration paths.
 source-paths: ["models"]
-# source-paths: ["test_models"] # TO DO: Configure via scripts for integration testing
 analysis-paths: ["analysis"]
 test-paths: ["tests"]
 data-paths: ["data"]
@@ -21,15 +21,6 @@ clean-targets:
     - "target"
     - "dbt_modules"
 
-# Configured for the logger macro.
+# Configured for the dbt_meta_testing.logger macro.
 vars:
     logging_level: INFO
-
-# TO DO: Configure via scripts for integration testing
-# models:
-#   dbt_meta_testing:
-#       # Applies to all files under models/example/
-#       example:
-#           +materialized: view
-#           +required_tests: {'unique': 1, 'not_null': 2, 'mock_schema_test': 1}
-#           +required_docs: true
diff --git a/integration_tests/.gitignore b/integration_tests/.gitignore
@@ -0,0 +1,4 @@
+
+target/
+dbt_modules/
+logs/
-Original file line number
+Diff line change
@@ Expand Up / @@ -4,4 +4,4 @@ dbt_modules/ @@
     logs/
     .env
     .user.yml
-    profiles.yml
+    .vscode