Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot create external tables with partitions and non-default file format #99

Closed
jazumaquero opened this issue Apr 28, 2023 · 1 comment

Comments

@jazumaquero
Copy link

Hi,

Thanks for developing this dbt adapter for Hive,

I have being suffering some issues while trying to create external partitions tables with some specific file formats , for instance using next configuration

{{
  config(
    materialized="incremental",
    incremental_strategy= "insert_overwrite",
    external=True,
    file_format= "orc",
    partition_by=["some colum"],
  ) 
}}

Looks like this issue is happening because when the file format is added when table is being created (

{% else %}
create {% if is_external == true -%}external{%- endif %} table {{ relation }}
{% endif %}
{{ file_format_clause() }}
{{ options_clause() }}
{{ partition_cols(label="partitioned by") }}
{{ clustered_cols(label="clustered by") }}
{{ location_clause() }}
{{ comment_clause() }}
{{ properties_clause(_properties) }}
as
{{ sql }}
{%- endif %}
)

This is not consistent with Hive CREATE TABLE definition that expect STORED AS TO BE after partitioning and before location.

So that, I guess it should be something like following

    {% else %}
      create {% if is_external == true -%}external{%- endif %} table {{ relation }}
    {% endif %}
    {{ options_clause() }}
    {{ partition_cols(label="partitioned by") }}
    {{ clustered_cols(label="clustered by") }}
    {{ file_format_clause() }}
    {{ location_clause() }}
    {{ comment_clause() }}
    {{ properties_clause(_properties) }}
    as
      {{ sql }}

Best Regards,
Zuma

@niteshy
Copy link
Collaborator

niteshy commented Aug 10, 2023

This is resolved in #96

@niteshy niteshy closed this as completed Aug 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants