Fix #1718 (#1719)

* [Infra] Version 2.0.0b16 (#1678)

* [infra] Version 1.7.0 python-package

* [infra] fix update_columns test

* [infra] remove unused import

* [infra] add to_partition utility function

* [infra] add test for to_partitions

* [infra] pump package version 1.6.9-b2

* [infra] add break_file feature

* Revert "[infra] pump package version 1.6.9-b2"

This reverts commit 0cba449.

* feat: add `connection_id` to external data configuration

* fix(Datatype): add connection id for external configuration

* feat: add automatic management of BQ connection

* chore: fix linting issues

* feat: add test folder to gitignore

* feat: release beta version

* feat(Connection): add `service_account` property

* feat(Base): add IAM stuff

* chore: fix linting issues

* feat: automatic granting roles to BigLake service account

* feat: better error handling, set biglake permissions is now optional

* feat: release beta version

* chore: modify log message

* chore: make all partitions string

* chore: merge master

* add __version__ atribute (#1488)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* add option to change copied table name (#1489)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* fix: pylint

* changing python_path fixture

* adding shapely as dependency to downgrade if already installed, like Colab

* pylinting files

* bump version 1.6.10-beta.1

* bump version 1.6.10

* updating version

* return update_columns to Table class

* authentication methods in base class

* method to return dataset id from slug using graphql

* method to return table id from slug of dataset and table using graphql

* using variables in graphql query

* change default & log downloaded path

* authentication with graphql

* change version in pyproject.toml

* chore: refactor connection imports and make working dir default for storage download

* chore: make staging the default mode for storage download

* chore: make staging the default mode for storage download

* fix: pylint

* chore: release new beta version

* small corrections in 1.6.11

* methods to retrieve metadata from graphql api

* adding data to api_data_dict

* adding exists_in_api method

* changing is_updated method

* method to get a request in graphql

* logging errors with loguru, instead of print

* writing yaml files before updating the database

* helper to convert case from snake_case to camelCase and vice versa

* start refactoring the query to use alias

* refactoring: clean edges and nodes from graphql response

* moving graphql queries for separated files and others

* improving unit tests and graphql for api_metadata

* changing api_response for compatibility with current yamls

* Hotfix storage init args (#1576)

* [dados]br_ibge_estadic.indicadores (#1535)

* dados

* dados

* title

* observation_level

* escolaridade

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [dados] br_ipea_avs (#1530)

* up br_ipea_avs

* Ajeitando os comentários da equipe de dados.

* Alterando a temporal_coverage

* Ajustando o PR.3

* Create code

* Delete code

* Create br_avs_ipea

* Add files via upload

* Delete br_avs_ipea

* Subindo novamente toda a base, devido as alterações.

* Ajustando o PR

* Delete br_ipea_avs.ipynb

* update

* update

* update

* update

* update

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Update table_config.yaml (#1553)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [dados] br_me_clima_organizacional (#1548)

* dados

* partner_organization

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [dados] br_me_exportadoras_importadoras  (#1521)

* Sobe br_me_exportadoras_importadoras

* Faz correções no script

* Corrige erros no table_config apontados na correção do PR

* Corrige erros no table_config apontados na correção do PR v2

* Corrige erros no dicionário

* Corrige tipo do CEP

* Corrige o tipo do dado do CEP no publish.sql

* Delete dataset_config yamls

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Laura Amaral <100051996+laura-l-amaral@users.noreply.github.com>

* [dados] world_fao_production (#1536)

* Abrir PR world_fao_production

* Corrige erros apontados na correção do PR

* Corrige erros apontados na correção do PR v2

* Altera nível de observação do table_cofing da tabela item

* Corrige erros apontados na correção do PR

* Corrige a partição e altera a nome da variável ano para year

* remove palavra repetida da descrição

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Laura Amaral <100051996+laura-l-amaral@users.noreply.github.com>

* [dados] world_wb_wwbi.country_finance (#1538)

* dados

* update

* data

* add observacoes

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [dados-atualizacao] update pib municipio (#1559)

* update pib municipio

* updates

* updates

* Update table_description.txt

---------

Co-authored-by: Gabrielle Carvalho <77730866+gabrielle-carv@users.noreply.github.com>

* [dados] br_ibge_estadic (#1560)

* update

* update metadata_modified

* update metadata_modified

* update metadata_modified

---------

Co-authored-by: Laura Amaral <100051996+laura-l-amaral@users.noreply.github.com>

* [dados] br_ibge_munic (#1534)

* dados

* update table_config

* update

* code

* code novo

* sigla_uf

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [dados] br_me_siconfi.uf (#1546)

* upload br_me_siconfi_uf

* update review gabs

* Update table_config.yaml

* Update table_config.yaml

* Update table_config.yaml

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Laura Amaral <100051996+laura-l-amaral@users.noreply.github.com>

* [dados] br_ipea_avs (#1564)

* up br_ipea_avs

* up br_ibge_estadic

* Ajustando os comentários da equipe de dados.

* Ajustando cobertura temporal.

* Delete README.md

* Delete dataset_config.yaml

* Delete publish.sql

* Delete schema-prod.json

* Delete schema-staging.json

* Delete table_config.yaml

* Delete table_description.txt

* update

* update

* update

* update

* update

* update

* update

* Delete br_ibge_estadic_educação.ipynb

apagando código estadic

* Delete publish.sql

apagando publish.sql

* Delete schema-prod.json

apagando schema-prod

* Delete schema-staging.json

apagando schema-staging

* Delete table_config.yaml

apagando table_config

* Delete table_description.txt

apagando table_description

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [dados] br_bcb_estban (#1561)

* Abre PR do conjunto br_bcb_estban

* Corrige erros apontados na correção do PR

* Delete dataset_config.yaml

* Corrige erros apontados na correção do PR v2.0

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Laura Amaral <100051996+laura-l-amaral@users.noreply.github.com>

* update dicionario br_me_rais (#1567)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [dados] br_ibge_estadic (#1531)

* up br_ipea_avs

* up br_ibge_estadic

* Ajustando os comentários da equipe de dados.

* Ajustando cobertura temporal.

* Delete README.md

* Delete dataset_config.yaml

* Delete publish.sql

* Delete schema-prod.json

* Delete schema-staging.json

* Delete table_config.yaml

* Delete table_description.txt

* update

* update

* update

* update

* update

* update

* update

* update

* update dicionario br_ibge_estadic

* update br_ibge_estadic

* update dicionário br_ibge_estadic

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [dados-atualizacao] update `br_sgp_informacao.despesas_cartao_corporativo` (#1570)

* update cartao corporativo

* Update table_config.yaml

* [dados] world_spi (#1555)

* update world_spi

* update

* update

* update_2

* update world_spi

* fix: dataset_id

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Crislane Alves <58278652+crislanealves@users.noreply.github.com>

* add required args

* [dados-atualizacao] br_inep_indicadores_educacionais (#1566)

* [dados-atualizacao]

atualiza os dados para 2022; atualiza table_config, cria script em python

* Update table_config.yaml

* Update table_config.yaml

* Update table_config.yaml

* Update table_config.yaml

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

---------

Co-authored-by: Gabrielle Carvalho <77730866+gabrielle-carv@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Patrick Teixeira <105399231+tricktx@users.noreply.github.com>
Co-authored-by: Gabriel Pisa <61624649+folhesgabriel@users.noreply.github.com>
Co-authored-by: Laura Amaral <100051996+laura-l-amaral@users.noreply.github.com>
Co-authored-by: Arthur Gusmão <arthurgusmao@id.uff.br>
Co-authored-by: Crislane Alves <58278652+crislanealves@users.noreply.github.com>
Co-authored-by: Fernanda Scovino <fscovinom@gmail.com>
Co-authored-by: Lucas Moreira <65978482+lucasnascm@users.noreply.github.com>

* removing coverage from query

* fixes in dataset and table config files creation

* owner_org and exists_in_api methods

* removing references to REST API and treat errors in login

* adjustments in data_dict, tests for publish

* initializing RemoteAPI for mutations

* mutation to create a dataset

* adjustments in data_dict and others

* corrections and new table for publish tests

* removing IDE settings from project

* prevent exclusion of tmp_bases when it already exists

* change filename to avoid conflict in tests

* commiting notebook file as it does not rollback, despite the fact that contents are identical

* stable exists method

* metadata_modified as datetime

* chore: remove yaml dependecy from metadata.py

* chore: remove yaml dependecy from metadata.py

* feat: remove more code

* feat: remove more code

* initial structure

* chore: refactor graphql requests

* chore: refactor graphql requests

* chore: refactor graphql requests

* chore: make publish_sql

* feat: add backend class for handling interaction with graphql

* chore: clean some code and comment parts where table_config are needed

* chore: clean some code and comment parts where table_config are needed

* chore: add dataset config query

* chore: add table config query

* feat: create dataset and use API metadata

* chore(deps): remove unnecessary deps

* chore: minor cleanup

* chore: delete file

* feat: fix occurences of `table_config`

* feat: add API url to config init

* feat: add structure for `Metadata.create`

* chore: more table modifications

* feat: table create using data_columns and partitioned data

* feat: some refactor and finish table.create

* chore: better casing

* chore: better casing

* chore: better logging

* chore: update table.create docstring

* chore: clean config files

* feat: refactor table.publish and table.update

* chore: make publish.sql from staging schema

* feat: get partition dict from storage

* chore: rename some methods

* chore: update and publish only acts in prod and uses the staging table schema to generate the prod publish query and update schema

* chore: load schema using SchemaField, remove code that depends on template

* chore: refactor init process

* chore: remove upload function from cli

* chore: remove upload function from cli

* chore: clean unused imports, redo poetry packages and release 2.0.0-b1

* chore: add a new dependencie requests-toolbelt

* chore: add tomlkit and better error if columns does not have name

* chore: error handling and  make publish and update get info from api if existis

* fix: typo in _get_columns_from_data and better infos

* chore: add tomlkit

* chore: error handling in case that API is off

* chore: error handling in case that API is off

* hotfix: chang metadata base_url

* chore: get backend metadata from cloud tables

* feat: bump beta version

* chore: change mode in table.delete

* chore: pump version

* chore: no more version number on files

* feat: implement external warnings and messages

* feat: add csv_delimiter and allow csv_allow_jagged_rows

* fix: pump bd version and add new parameters csv_delimiter and csv_allow_jagged_rows

* chore: cleanup

* chore: remove compressed r package

* chore: refactor dependency management

* chore: fix linting issues

* chore: remove pylint action

* fix: change install instructions

* feat: pump bd version

* fix: change install instructions

* feat: pump bd version

* feat: add new parameter csv_skip_leading_rows and setup.py

* feat: pump bd version

* chore: fix conflicts

* chore: add `all` extra

* chore: lint

* feat: create branch v2.0.0

* fix: add csv delimiter to schema

* feat: expand credential scope to drive and bq

* chore: start cleaning tests

* chore: add timeout to pypi warning

---------

Co-authored-by: lucascr91 <lucas.ecomg@gmail.com>
Co-authored-by: Mauricio Fagundes <mauricio.fagundes@gmail.com>
Co-authored-by: Gabriel Gazola Milan <gabriel.gazola@poli.ufrj.br>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Fernanda Scovino <fscovinom@gmail.com>
Co-authored-by: Gabrielle Carvalho <77730866+gabrielle-carv@users.noreply.github.com>
Co-authored-by: Patrick Teixeira <105399231+tricktx@users.noreply.github.com>
Co-authored-by: Gabriel Pisa <61624649+folhesgabriel@users.noreply.github.com>
Co-authored-by: Laura Amaral <100051996+laura-l-amaral@users.noreply.github.com>
Co-authored-by: Arthur Gusmão <arthurgusmao@id.uff.br>
Co-authored-by: Crislane Alves <58278652+crislanealves@users.noreply.github.com>
Co-authored-by: Lucas Moreira <65978482+lucasnascm@users.noreply.github.com>

* chore: remove cli and its dependencies (#1707)

* chore: update lint and workflows (#1708)

* chore: remove deprecated git hooks

* chore: update github templates

* chore: update github workflows

* chore: update linters and pre-commit hooks

* chore: update readme (#1709)

* chore: remove contributors as we already removed the workflow

* chore: update readme getting started links

* chore: remove pylint comments (#1710)

* chore: remove schemas (#1711)

* feat: add containers to docs development (#1713)

* feat: conform br_tse_eleicoes code to DBT tests

* feat: update backend and metadata modules (#1712)

* chore: move config templates to templates folder

* feat: update backend and metadata modules

* updates to br_tse_eleicoes, br_me_rais

* updates to br_tse_eleicoes, br_me_rais

* small improvements

* Close #1718

* remove overwrite_cli_config

---------

Co-authored-by: Diego Oliveira <d116626@gmail.com>
Co-authored-by: lucascr91 <lucas.ecomg@gmail.com>
Co-authored-by: Mauricio Fagundes <mauricio.fagundes@gmail.com>
Co-authored-by: Gabriel Gazola Milan <gabriel.gazola@poli.ufrj.br>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Fernanda Scovino <fscovinom@gmail.com>
Co-authored-by: Gabrielle Carvalho <77730866+gabrielle-carv@users.noreply.github.com>
Co-authored-by: Patrick Teixeira <105399231+tricktx@users.noreply.github.com>
Co-authored-by: Gabriel Pisa <61624649+folhesgabriel@users.noreply.github.com>
Co-authored-by: Laura Amaral <100051996+laura-l-amaral@users.noreply.github.com>
Co-authored-by: Arthur Gusmão <arthurgusmao@id.uff.br>
Co-authored-by: Crislane Alves <58278652+crislanealves@users.noreply.github.com>
Co-authored-by: Lucas Moreira <65978482+lucasnascm@users.noreply.github.com>
Co-authored-by: Vinicius Aguiar <vncsna@gmail.com>
Co-authored-by: Ricardo Dahis <rdahis@gmail.com>

python-package/basedosdados/upload/table.py

@@ -561,7 +561,12 @@ def create(
                 Path,
             ),
         ):
-            Storage(self.dataset_id, self.table_id).upload(
+            Storage(
+                dataset_id=self.dataset_id,
+                table_id=self.table_id,
+                config_path=self.config_path,
+                bucket_name=self.bucket_name,
+            ).upload(
                 path=path,
                 mode="staging",
                 if_exists=if_storage_data_exists,