How to upload new project with two factor auth enabled?

[grantjenks]

Describe the bug

I want to upload a new project and my account has two factor auth enabled. When I try to upload locally with twine, I get an error saying that I cannot use my password because I have two factor auth enabled. Then I try uploading from GitHub using the new OIDC functionality but I get another error saying that non-user accounts cannot create new projects.

Expected behavior

There is a way to upload to PyPI with a two factor auth account.

To Reproduce

GitHub action log: https://github.com/grantjenks/python-typesense-server-wrapper/actions/runs/5480180242/jobs/9983014023

My Platform

Python 3.11 from GitHub Codespaces and GitHub Actions on ubuntu-latest.

Additional context

• https://stackoverflow.com/questions/57506394/how-to-upload-package-to-pypi-with-two-factor-enabled Says to use api tokens but I can’t create those for a new project

• https://stackoverflow.com/questions/62532237/how-can-i-create-an-api-token-on-pypi-for-a-new-project/62535186#62535186 Says that I need to create a global token and swap it later but that doesn’t seem right

• Single-use API token to create new package #6378 Is closed but the resolution is unclear to me. The resolution there seems to be to use OIDC but that didn’t work for me.

[di]

Hi @grantjenks, sorry you're having trouble and thanks for the report. The two ways to create projects when 2FA is enabled are:

• Use an account-scoped API token rather than a project-scoped API token: at https://pypi.org/manage/account/token/, you can select "Entire account (all projects)" to generate an API token that can create new projects.
• Use OIDC / "Trusted Publishing" (preferred): https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/

This might be a bug on our end: I see that you correctly added a "pending publisher" for https://github.com/grantjenks/python-typesense-server-wrapper.

I also see that during the workflow run, https://pypi.org/p/python-typesense-server-wrapper/ actually did get created and the pending publisher was correctly converted it to an actual publisher (you can see it here: https://pypi.org/manage/project/python-typesense-server-wrapper/settings/publishing/). However, no releases were published.

In this case, the token exchange prior to the upload successfully created the project, but by the time the request to upload the releases was made, the following request thought the project didn't exist for some reason, and so it failed:

warehouse/warehouse/forklift/legacy.py

Lines 946 to 958 in 30de226

	if project is None:
	# Another sanity check: we should be preventing non-user identities
	# from creating projects in the first place with scoped tokens,
	# but double-check anyways.
	if not request.user:
	raise _exc_with_message(
	HTTPBadRequest,
	(
	"Non-user identities cannot create new projects. "
	"You must first create a project as a user, and then "
	"configure the project to use trusted publishers."
	),
	)

I'm not sure what's happening here: it's possible the two requests happened too quickly and the creation of the project wasn't committed to our database before the upload request happened. At any rate, the error message is wrong and potentially confusing, so we should look into improving that as well.

The good news is you should be able to re-run that same workflow and it should work successfully this time.

(cc @webknjaz @woodruffw for any thoughts they might have)

[woodruffw]

Yeah, this is odd -- to have gotten that far request.identity must be non-None which in turn means it must be a valid publisher identity, so the corresponding project should have been created and made available at that point.

Agreed about the error message being confusing though; I'll give some thought to what a better one would be.

[webknjaz]

That assessment seems fair. Too bad we can't reliably know if it's safe to call twine, otherwise sticking a retry would've helped on the action side...

[di]

Too bad we can't reliably know if it's safe to call twine

Can you clarify what you mean by this?

[grantjenks]

Thanks for the quick responses, all! I tried re-running the workflow but failed the same way. Log at https://github.com/grantjenks/python-typesense-server-wrapper/actions/runs/5480180242/jobs/9985978215

Anything else I can do to get the GHA working?

[di]

Aha, @grantjenks, should the project name be typesense_server_wrapper or python-typesense-server-wrapper? Looks like you've configured the latter on the PyPI publisher, but the former for the project itself.

[webknjaz]

Too bad we can't reliably know if it's safe to call twine

Can you clarify what you mean by this?

When we run twine upload from the action, there's no way to act on its error message. This is because there's no API and we can only see if the entire operation succeeded or failed. With a failure, it could've succeeded uploading a few artifacts but crashed on the rest, for example.
If twine were provide these details, it'd make it possible for the action to attempt another upload safely.
Also, we can't even display the information and recommendations in better places since twine only outputs to the console and that's about it..
I wish twine had better integration capabilities, it would open up some possibilities of being more helpful to the end-users with better explanations and hints in GHA...

[webknjaz]

Looks like you've configured the latter on the PyPI publisher, but the former for the project itself.

This must be it. If these don't match, PyPI can't find a publisher trust connection for that... I wonder if PyPI should silently normalize the project name both on creation and upload.. cc @woodruffw

Also, I'd argue that the error message could be improved — just by reading the log, I couldn't understand why it's talking about some service identity. And that's me with likely better understanding of the context than an average user. Can we provide simpler explanation there?
Of course, we could attempt giving a hint on the action side but adding a new hint for every corner case feels wasteful..

[di]

I wonder if PyPI should silently normalize the project name both on creation and upload.. cc @woodruffw

@webknjaz We already normalize the project name, but even with normalization, the project names here aren't the same.

Also, I'd argue that the error message could be improved

Yes, looks like this is an edge case we hadn't considered. I think at this point we have enough information to provide an error message like:

You attempted to upload to {some project name}, but the publisher you used is configured for {some other project name}.

[woodruffw]

Yep, agreed with @di -- I can make a PR to improve this error message in a moment.

[webknjaz]

Ah, I missed that they differ more..

@woodruffw how about improving the form on PyPI with extra explanation that the name shouldn't come from the repo slug but from something like pyproject.toml?

[woodruffw]

@woodruffw how about improving the form on PyPI with extra explanation that the name shouldn't come from the repo slug but from something like pyproject.toml?

Yeah, I'll think about the language there -- we might not want to reference specific sources of metadata since PyPI is agnostic to whether it's coming from pyproject.toml or some other source.

Either way, I think we should add this as a troubleshooting item to the trusted publisher docs as well. I'll include that in the error message work.

[woodruffw]

#14082 improves the error here and adds a corresponding troubleshooting section.

[grantjenks]

Thanks all. I fixed the name on PyPI and the job on GitHub worked. Sorry for the mistake.

[woodruffw]

Thanks all. I fixed the name on PyPI and the job on GitHub worked. Sorry for the mistake.

Nothing to be sorry for -- this is a reasonable source of confusion, and you helped us find and fix a misleading error message!