Skip to content

Commit

Permalink
Update databricks-changes.md
Browse files Browse the repository at this point in the history
Editorial changes
  • Loading branch information
matthewshaver authored Oct 29, 2024
1 parent a868ce8 commit 603ef22
Showing 1 changed file with 7 additions and 10 deletions.
17 changes: 7 additions & 10 deletions website/docs/reference/global-configs/databricks-changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,7 @@ id: "databricks-changes"
sidebar: "Databricks"
---

## Behavior Changes

Here are the current [behavior change flags](/docs/reference/global-configs/behavior-changes.md#behavior-change-flags) that are specific to `dbt-databricks`:
The following are the current [behavior change flags](/docs/reference/global-configs/behavior-changes.md#behavior-change-flags) that are specific to `dbt-databricks`:

| Flag | `dbt-databricks`: Intro | `dbt-databricks`: Maturity |
| ----------------------------- | ----------------------- | -------------------------- |
Expand All @@ -16,14 +14,13 @@ Here are the current [behavior change flags](/docs/reference/global-configs/beha
### Use information schema for columns

The `use_info_schema_for_columns` flag is `False` by default.
Setting this flag to `True` will use `information_schema`, rather than `describe extended`, to get column metadata for Unity Catalog tables.
This setting is to avoid issues where `describe extended` truncates type information when the type is a complex struct. However, this setting is not ready to be the default behavior, as there is a performance impact due to a Databricks metadata limitation. This is due to
the need to run `REPAIR TABLE {{relation}} SYNC METADATA` prior to querying in order to ensure the `information_schema` is complete.

Setting this flag to `True` will use `information_schema` rather than `describe extended` to get column metadata for Unity Catalog tables. This setting helps you avoid issues where `describe extended` truncates information when the type is a complex struct. However, this setting is not yet the default behavior, as there are performance impacts due to a Databricks metadata limitation because of the need to run `REPAIR TABLE {{relation}} SYNC METADATA` before querying to ensure the `information_schema` is complete.

This flag will become the default behavior when this additional query is no longer needed.

### Use user's folder for Python model notebooks

The `use_user_folder_for_python` flag is `False` by default, and results in writing uploaded python model notebooks to `/Shared/dbt_python_models/{{schema}}/`.
Setting this flag to `True` will instead write notebooks to `/Users/{{current user}}/{{catalog}}/{{schema}}/`
Writing to the `Shared` folder is being deprecated by Databricks as it does not align with governance best practices.
We plan to promote this flag to maturity in 1.10.0.
The `use_user_folder_for_python` flag is `False` by default and results in writing uploaded python model notebooks to `/Shared/dbt_python_models/{{schema}}/`. Setting this flag to `True` will write notebooks to `/Users/{{current user}}/{{catalog}}/{{schema}}/` Writing to the `Shared` folder is deprecated by Databricks as it does not align with governance best practices.

We plan to promote this flag to maturity in v1.10.0.

0 comments on commit 603ef22

Please sign in to comment.