Compute Information in Target Information for Jinja templating

[RJaikanthEJ]

Hi,

I am trying to create a macro that defines different SQL warehouses for different models/environments. For ex - dev would almost always have 2x-small and production x-large. It can also be that different models require different warehouses depending on size.

I am looking for something similar to - dbt-labs/dbt-snowflake#103 (comment) but for the databricks adapter. I know we can set compute manually from https://docs.getdbt.com/reference/resource-configs/databricks-configs#selecting-compute-per-model but looking for a way to automate based on environment first and table size in the next iteration.

I could not find any documentation about what information can be accessed via the target variable. It would be helpful if you can point towards that.

[benc-db]

This is already possible without a macro. The compute definitions are per 'output' in your profile, so just use the same tag with a different definition that's matched to each env.
ex:

my_profile:
  target: dev
  outputs:
    dev:
      type: databricks
      ...
      compute:
        small_tables:
          http_path: /sql/path/for/small_tables_in_dev

        big_tables:
           http_path: /sql/path/for/big_tables_in_dev

    prod:
      type: databricks
      ...
      compute:
        small_tables:
          http_path: /sql/path/for/small_tables_in_prod

        big_tables:
           http_path: /sql/path/for/big_tables_in_prod

[RJaikanthEJ]

Hi @benc-db

Thanks for the answer it is really helpful. Just for any future use case, is there a way to access the compute using a macro. Would we do it like target.compute.small_tables.http_path ?

[benc-db]

That's a good question that I don't know the answer to off the top of my head. I recommend experimenting with a SQL comment like:

-- {{ target.compute.small_tables.http_path }}

and see what shows up in the dbt.log. Also, someone on the dbt slack may know.