Skip to content

Viadot 0.4.0
Compare
Choose a tag to compare
@m-paz m-paz released this 07 Apr 12:44
· 1835 commits to main since this release
v0.4.0
1e281f6
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Added

  • Added custom_mail_state_handler function that sends mail notification using custom smtp server.
  • Added new function df_clean_column that cleans data frame columns from special characters
  • Added df_clean_column util task that removes special characters from a pandas DataFrame
  • Added MultipleFlows flow class which enables running multiple flows in a given order.
  • Added GetFlowNewDateRange task to change date range based on Prefect flows
  • Added check_col_order parameter in ADLSToAzureSQL
  • Added new source ASElite
  • Added KeyVault support in CloudForCustomers tasks
  • Added SQLServer source
  • Added DuckDBToDF task
  • Added DuckDBTransform flow
  • Added SQLServerCreateTable task
  • Added credentials param to BCPTask
  • Added get_sql_dtypes_from_df and update_dict util tasks
  • Added DuckDBToSQLServer flow
  • Added if_exists="append" option to DuckDB.create_table_from_parquet()
  • Added get_flow_last_run_date util function
  • Added df_to_dataset task util for writing DataFrames to data lakes using pyarrow
  • Added retries to Cloud for Customers tasks
  • Added chunksize parameter to C4CToDF task to allow pulling data in chunks
  • Added chunksize parameter to BCPTask task to allow more control over the load process
  • Added support for SQL Server's custom datetimeoffset type
  • Added AzureSQLToDF task
  • Added AzureSQLUpsert task

Changed

  • Changed the base class of AzureSQL to SQLServer
  • df_to_parquet() task now creates directories if needed
  • Added several more separators to check for automatically in SAPRFC.to_df()
  • Upgraded duckdb version to 0.3.2

Fixed

  • Fixed bug with CheckColumnOrder task
  • Fixed OpenSSL config for old SQL Servers still using TLS < 1.2
  • BCPTask now correctly handles custom SQL Server port
  • Fixed SAPRFC.to_df() ignoring user-specified separator
  • Fixed temporary CSV generated by the DuckDBToSQLServer flow not being cleaned up
  • Fixed some mappings in get_sql_dtypes_from_df() and optimized performance
  • Fixed BCPTask - the case when the file path contained a space
  • Fixed credential evaluation logic (credentials is now evaluated before config_key)
  • Fixed "$top" and "$skip" values being ignored by C4CToDF task if provided in the params parameter
  • Fixed SQL.to_df() incorrectly handling queries that begin with whitespace

Removed

  • Removed autopick_sep parameter from SAPRFC functions. The separator is now always picked automatically if not provided.
  • Removed dtypes_to_json task to task_utils.py
Assets 2
Loading