Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: refine constraint to avoid numpy pandas incompatability #277

Merged
merged 2 commits into from
Jun 27, 2024

Conversation

chalmerlowe
Copy link
Collaborator

@chalmerlowe chalmerlowe commented Jun 27, 2024

Pins the version of numpy used with python 3.9 to be less than 2.0.0.

There is a potentially unexpected interaction between pandas and numpy here. Depending on the version we use for each, we get different results in terms of pass/fail for the unit tests associated with python 3.9

numpy version pandas version unit-3.9 pass/fail
2.0.0 1.3.0 fail
2.0.0 2.2.2 pass
2.0.0 1.5.3 fail
1.26.4 1.5.3 pass
1.26.4 1.3.0 pass

This appears to be a known incompatibility between how pip resolves the dependency versions for numpy and pandas and the only current fix is to pin versions to something that "works" to avoid allowing pip to resolve into a failing state. (NOTE the linked issue references various versions of numpy and pandas, depending on the combination, inlcuding 3.9 throughout the thread, even though the title references python 3.12)

Fixes #275 🦕

@chalmerlowe chalmerlowe requested review from a team as code owners June 27, 2024 15:38
@chalmerlowe chalmerlowe requested a review from alvarowolfx June 27, 2024 15:38
Copy link

conventional-commit-lint-gcf bot commented Jun 27, 2024

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot
https://conventionalcommits.org/

@product-auto-label product-auto-label bot added size: xs Pull request size is extra small. api: bigquery Issues related to the googleapis/python-db-dtypes-pandas API. labels Jun 27, 2024
@chalmerlowe chalmerlowe mentioned this pull request Jun 27, 2024
@chalmerlowe chalmerlowe added the automerge Merge the pull request once unit tests and other checks pass. label Jun 27, 2024
Copy link
Contributor

@Linchin Linchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I have two questions:

  1. Why don't we pin pandas to be 2.2.2 instead?
  2. Owlbot made some change. Maybe we will need to revert that and add it to the owlbot.py

@gcf-merge-on-green gcf-merge-on-green bot merged commit 55dff95 into main Jun 27, 2024
27 checks passed
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Jun 27, 2024
@gcf-merge-on-green gcf-merge-on-green bot deleted the clowe-dtype-size-difference branch June 27, 2024 17:34
@tswast
Copy link
Collaborator

tswast commented Aug 5, 2024

Why don't we pin pandas to be 2.2.2 instead?

Ideally, we'd spread out the versions of pandas we test. For pinning one of the newer versions of pandas, that should be on one of the later Python versions.

Note that currently, we test on:

Ideally, we would test on multiple 2.x versions but continue to have the latest supported Python test with latest supported pandas.

Also, we might want to consider bumping our minimum pandas to 1.5.x sometime. That's the latest 1.x release and has the most flexibility of 1.x versions with regards to extension dtypes and pyarrow support.

Copy link
Collaborator

@tswast tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-db-dtypes-pandas API. size: xs Pull request size is extra small.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The build failed
3 participants