Skip to content

Commit

Permalink
fix ephemeral issue, fix models errors, add gitignore (#6)
Browse files Browse the repository at this point in the history
* fix ephemeral issue, fix models errors, add gitignore

* updated readme, limited ephemeral checks to required_docs

* touch notes on README

* revert fetch_configured_models
  • Loading branch information
tnightengale authored May 3, 2021
1 parent a3f60f9 commit b360f5f
Show file tree
Hide file tree
Showing 4 changed files with 9 additions and 44 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ logs/
.env
.user.yml
.vscode
.dbt_env
48 changes: 6 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ This dbt package contains macros to assert test and documentation coverage from
- [Usage](#usage)
- [required_tests (source)](#required_tests-source)
- [required_docs (source)](#required_docs-source)
- [dbt_meta_testing.logger (source)](#dbt_meta_testinglogger-source)
- [Contributions](#contributions)
- [Testing](#testing)
- [Verified Data Warehouses](#verified-data-warehouses)
Expand Down Expand Up @@ -115,9 +114,12 @@ models:
project:
+required_docs: true
```
The `+required_docs` config must be a `bool`.
The `+required_docs` config must be a `bool`. It also **does not check ephemeral
models**. This is because it cannot leverage `adapter.get_columns_in_relation()`
macro on ephemeral models, which it uses to fetch columns from the data
warehouse and detect columns without documentation.

When applied to a model, this config will ensure 3 things:
When applied to a non-ephemeral model, this config will ensure 3 things:
1. The _model_ has a non-empty description
2. The _columns_ in the model are specified in the model `.yml`
3. The _columns_ specified in the model `.yml` have non-empty descriptions
Expand Down Expand Up @@ -236,45 +238,7 @@ Usage:
$ dbt run-operation required_docs [--args "{'models': '<space_delimited_models>'}"]
```
**Note:** Run this command _after_ `dbt run`: only models that already exist in
the warehouse can be validated for columns that are missing from the model
`.yml`.

### dbt_meta_testing.logger ([source](macros/logger.sql))
An ammenity macro that mimics pythons logging module. The level is passed via
the `log_level` kwarg. The default "method" (ie. `log_level`) is `DEBUG`,
similar to calling `logging.debug('some buggy thing')` in python.


Set a level with a project var, similar to
`logging.basicConfig(level=logging.INFO)` in python:
```yaml
# dbt_project.yml
...
vars:
logging_level: WARNING
```
The default `logging_level` is `INFO`.

Usage:
```sql
-- macros/some_macro.sql
...
{% if my_object is mapping %}
-- This will be a DEBUG call by default, it won't show up if logging_level="INFO" or higher
{{ dbt_meta_testing.logger("my_object is mapping: " ~ my_object is mapping) }}
...
{% endif %}
-- This will be an INFO call, it will show up if logging_level="INFO" or lower
{{ dbt_meta_testing.logger("The following keys were found: " ~ my_complex_object.keys(), log_level="INFO") }}
{{ return(my_complex_object) }}
```
the warehouse can be validated for columns that are missing from the model `.yml`.

## Contributions
Feedback on this project is welcomed and encouraged. Please open an issue or
Expand Down
2 changes: 1 addition & 1 deletion macros/utils/errors/error_required_docs.sql
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
) %}

{% set all_errors = [] %}
{% if missing_description_errors | length > 0 %}
{% if missing_model_errors | length > 0 %}

{% do all_errors.append("The following models are missing descriptions:") %}
{% do all_errors.append(dbt_meta_testing.format_error_docs(missing_model_errors)) %}{% endif %}
Expand Down
2 changes: 1 addition & 1 deletion macros/utils/required_docs/evaluate_required_docs.sql
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

{% for model in models_to_evaluate %}

{% if model.config.required_docs==True %}
{% if model.config.required_docs==True and model.config.get("materialized", "") not in ("", "ephemeral")%}

{% set model_columns = adapter.get_columns_in_relation(ref(model.package_name, model.name))
| map(attribute="column") | list %}
Expand Down

0 comments on commit b360f5f

Please sign in to comment.