added macro for wh selection #503

McKnight-42 · 2023-03-07T20:35:15Z

resolves #438

Description

take over pr: #115 to get pre-commit issues taken care of and ready for merge

Checklist

I have read the contributing guide and understand what's expected of me
I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
I have run changie new to create a changelog entry

…on-macro' into mcknight/take-over-103

McKnight-42 · 2023-03-07T20:39:18Z

dbt/adapters/snowflake/impl.py

@@ -73,6 +76,8 @@ def _get_warehouse(self) -> str:

    def _use_warehouse(self, warehouse: str):
        """Use the given warehouse. Quotes are never applied."""
+        kwargs = {"warehouse": warehouse}
+        warehouse = self.execute_macro(SNOWFLAKE_WAREHOUSE_MACRO_NAME, kwargs=kwargs)  # type: ignore


curious if anyone has an idea as to why this causes a
dbt/adapters/snowflake/impl.py:80: error: Incompatible types in assignment (expression has type "AttrDict", variable has type "str") [assignment] error? seems same as all implementations in dbt-core I could find. just curious if there is better path than the ignore

It's silly, but you can remove the type annotation from warehouse and mypy will be happy; you can also update it to Any. The issue is that self.execute_macro() lists the type for kwargs as Dict[str, Any]. mypy doesn't like it when you make the type more strict (str is a proper subset of Any).

oh that's interesting iI didn't think the type value for the "warehouse" input would bother the previously defined one for execute_macro thank you for clearing that up @mikealfare

mikealfare · 2023-03-08T02:42:30Z

.changes/unreleased/Added-20220321-162832.yaml

@@ -0,0 +1,3 @@
+kind: Added


Is this a supported format? I feel like our process to roll up change logs might not like this. Based on the timestamp, it might be an older format.

This part might be from before some more recent changes to our changie logic so yeah can remove

…efinition.

…knight/take-over-103

manugarri · 2023-03-17T10:39:14Z

@McKnight-42 nice! after some trial and error I managed to get it working on prerelease v1.5.0b3 , is there any timeline to when is gonna be released on pypi?

McKnight-42 · 2023-03-17T14:09:05Z

@manugarri we did some releases yesterday actually https://pypi.org/project/dbt-snowflake/1.5.0b3/ I believe this should be apart of that release

manugarri · 2023-03-17T16:44:45Z

Awesome, great! One question I have, is it not possible to access environment variables inside this macro?

im trying to return different warehouses depending on the snowflake user running dbt, this is what i have so far

    {% set warehouses_specs = {
            "user1": {
                "SMALL": "wh1_small",
                "LARGE": "wh1_large"
            },
            "user2": {
                "SMALL": "wh2_small",
                "LARGE": "wh2_large"
            }
        }
    %}

    {{ return(warehouses_specs[env_var('SNOWFLAKE_USER')][warehouse] ) }}
{%- endmacro %}

however when i do dbt run i get these weird error:
'MacroManifest' object has no attribute 'env_vars'

How could I read the current_user inside this macro?

For what I can tell, it seems that you cant read env_vars from inside this macro.

For example, this macro definition throws the exception

{% macro snowflake_warehouse(warehouse) -%}
  {% set foo = env_var('SNOWFLAKE_USER') %}
  {{ return(warehouse) }}
{%- endmacro %}

while this one doesnt

{% macro snowflake_warehouse(warehouse) -%}
  {{ return(warehouse) }}
{%- endmacro %}

UPDATE: After a lot of help from dylan at the dbt slack channel, this particular use case (using the snowflake user to decide which warehouse to run) can be solved by using target.user.

However, there seem to be a bug with the implementation that makes it impossible to reference environment variables in this particular macro

This reverts commit 2e061fc.

McKnight-42 · 2023-03-24T16:17:59Z

after some more digging we have decided for now to actually revert #503 we now think that this method is the wrong way as it allows for some unintended consequences and will be trying to find a better way forward. sorry for any inconvenience this causes

* reverting #505 * Revert "added macro for wh selection (#503)" This reverts commit 2e061fc.

javiCalvo and others added 8 commits March 21, 2022 16:17

added macro for wh selection

82c640d

added changes

40c5948

Merge branch 'main' into feature/custom-warehouse-selection-macro

aa20ddc

Merge branch 'main' into feature/custom-warehouse-selection-macro

9c665d8

Merge branch 'main' into feature/custom-warehouse-selection-macro

7d67aa1

Merge branch 'main' into feature/custom-warehouse-selection-macro

10b63ca

Merge remote-tracking branch 'jc-103/feature/custom-warehouse-selecti…

f3e68d8

…on-macro' into mcknight/take-over-103

init push of takeover of pr:115

db1f7cb

McKnight-42 requested review from nssalian and mikealfare March 7, 2023 20:35

McKnight-42 requested a review from a team as a code owner March 7, 2023 20:35

McKnight-42 self-assigned this Mar 7, 2023

cla-bot bot added the cla:yes label Mar 7, 2023

McKnight-42 mentioned this pull request Mar 7, 2023

added macro for wh selection #115

Closed

3 tasks

McKnight-42 commented Mar 7, 2023

View reviewed changes

mikealfare reviewed Mar 8, 2023

View reviewed changes

remove original commmunity changie entry and remove a variable type d…

ed0c3a0

…efinition.

McKnight-42 requested a review from mikealfare March 8, 2023 17:32

Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake into mc…

f6ca3bf

…knight/take-over-103

mikealfare approved these changes Mar 8, 2023

View reviewed changes

McKnight-42 merged commit 2e061fc into main Mar 8, 2023

McKnight-42 deleted the mcknight/take-over-103 branch March 8, 2023 18:55

McKnight-42 mentioned this pull request Mar 13, 2023

[CT-333] Macro for warehouse selection #103

Open

manugarri mentioned this pull request Mar 20, 2023

[ADAP-386] [Bug] prerelease macro snowflake_warehouse unexpected behaviour #533

Closed

2 tasks

McKnight-42 mentioned this pull request Mar 23, 2023

reverting #505 #536

Merged

6 tasks

McKnight-42 added a commit that referenced this pull request Mar 23, 2023

Revert "added macro for wh selection (#503)"

f889685

This reverts commit 2e061fc.

McKnight-42 added a commit that referenced this pull request Mar 24, 2023

reverting #505 (#536)

aa04011

* reverting #505 * Revert "added macro for wh selection (#503)" This reverts commit 2e061fc.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added macro for wh selection #503

added macro for wh selection #503

McKnight-42 commented Mar 7, 2023 •

edited

Loading

McKnight-42 Mar 7, 2023

mikealfare Mar 8, 2023

McKnight-42 Mar 8, 2023

mikealfare Mar 8, 2023

McKnight-42 Mar 8, 2023

manugarri commented Mar 17, 2023 •

edited

Loading

McKnight-42 commented Mar 17, 2023

manugarri commented Mar 17, 2023 •

edited

Loading

McKnight-42 commented Mar 24, 2023 •

edited

Loading

added macro for wh selection #503

added macro for wh selection #503

Conversation

McKnight-42 commented Mar 7, 2023 • edited Loading

Description

Checklist

McKnight-42 Mar 7, 2023

Choose a reason for hiding this comment

mikealfare Mar 8, 2023

Choose a reason for hiding this comment

McKnight-42 Mar 8, 2023

Choose a reason for hiding this comment

mikealfare Mar 8, 2023

Choose a reason for hiding this comment

McKnight-42 Mar 8, 2023

Choose a reason for hiding this comment

manugarri commented Mar 17, 2023 • edited Loading

McKnight-42 commented Mar 17, 2023

manugarri commented Mar 17, 2023 • edited Loading

McKnight-42 commented Mar 24, 2023 • edited Loading

McKnight-42 commented Mar 7, 2023 •

edited

Loading

manugarri commented Mar 17, 2023 •

edited

Loading

manugarri commented Mar 17, 2023 •

edited

Loading

McKnight-42 commented Mar 24, 2023 •

edited

Loading