Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(transfer): add non-decorated shared transfer function #275

Merged
merged 2 commits into from
Jan 15, 2025

Conversation

ividito
Copy link
Contributor

@ividito ividito commented Jan 15, 2025

To avoid ValueErrors when calling transfer_data, refactor tasks to invoke a shared transfer function, rather than calling a task from another task (which caused issues with the ti parameter).

@ividito ividito requested a review from anayeaye January 15, 2025 15:56
Copy link
Contributor

@anayeaye anayeaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The payload transformation is running in dev sm2a and seems to be working as expected. We do see a data access permission issue from the ingest API which is unexpected but also the dev api is not intended to generate metadata for assets in production anyway.

For the promotion step I think we may want to ask users (and automated events) to add both
transfer: true and
target_bucket: veda-data-store or the instance production bucket equivalent in configuration for the promotion DAG. In this PR all defaults are for veda production. What do you think?
cc: @smohiudd @slesaad @botanical

@ividito ividito merged commit 37eb592 into dev Jan 15, 2025
3 checks passed
@ividito ividito deleted the fix/transfer-decorators branch January 15, 2025 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants