Models, routes and views for creating OIDC publishers (pypi#10753)

* warehouse/oidc: rough model skeleton

* warehouse/oidc: fix imports

* warehouse/migrations: add migration for OIDC models

* warehouse/migrations: reformat

* warehouse/oidc: add basic verification logic

* oidc/services: reduce clock skew leeway to 30s

* warehouse/oidc: refactor claim verification

* oidc/models: fill in missing properties

* warehouse/migrations: remove original OIDC migration

Add many-many project-provider association.

* warehouse: add OIDC migration, fix association

* warehouse: reformat

* warehouse: OIDC route/view skeleton work

* warehouse: form, view logic for adding OIDC providers

* manage/views: disable HTTP cache, add TODO

* warehouse: move oidc views to "publishing"

...and make it a sub-page for project management.

* warehouse: provider deletion routing

* warehouse: shore up constraints, better error flashes

* warehouse/migrations: rebase revision

* warehouse/templates: update OIDC language

Refer to OIDC providers as "OpenID Connect publishers"

* warehouse: OIDC rate limiting groundwork

* manage/views: clean up OIDC events

* warehouse: use GitHub token for API requests, when available

* oidc/forms: special casing for rate limiting

Record errors with Sentry.

* warehouse: split user/repo form inputs apart

* warehouse/templates: link to GitHub's OIDC docs

* oidc/models: remove actor from checked claims

* templates/email: add OIDC email templates

* warehouse: fix templates, add email sending logic

* warehouse: add an AdminFlag for OIDC control

* oidc/models: use set operators

* oidc/forms: exception driven handling for GitHub API errors

* warehouse: OIDC ratelimiting logic

Also some small HTML fixes.

* warehouse/locale: update translations

* warehouse: lintage

* templates/manage/settings: remove vestigial HTML

* warehouse: address feedback

* Simplify form handling
* Validate GitHub usernames against a regex
* Fix form error presentation

* manage/views: more feedback addressing

* Prevent an infoleak in a session flash
* Reword a confusing comment

* Update warehouse/manage/views.py

Co-authored-by: Dustin Ingram <di@users.noreply.github.com>

* manage/views: fixups

* warehouse: add "OIDC provider removed" emails

* oidc/forms: use GH org regex in callable validator body

* warehouse/locale: update translations

* tests, warehouse: begin writing unit tests

* More tests, restructure for testing

* tests: fill in GitHubProviderForm tests

* tests, warehouse: more tests, adaptations for testing

* tests: more manage/view tests

* tests, warehouse: ratelimit tests, fix bug

* tests: round out ratelimiting

* tests: more tests

* tests, warehouse: OIDC deletion tests

Also, gets some coverage for free by reusing a helper.

* tests, warehouse: fill in model checks

Accommodations for testing.

* oidc/models: type hints

* warehouse/locale: `make translations`

* tests, warehouse: site-wide OIDC feature flag

* warehouse: `make translations`

* treewide: route to 404 when OIDC is disabled

Enable OIDC by default for development environments; update tests.

* warehouse: `make translations`

* Update warehouse/templates/manage/publishing.html

Co-authored-by: Joachim Jablon <ewjoachim@gmail.com>

* oidc/{interfaces,services}: simplify API

* tests: update

* warehouse/migrations: rebase

* tests, warehouse: move ratelimit hit up

* warehouse: `make translations`

* warehouse: plug in more OIDC metrics

Adds additional metrics on:

* Publisher configuration (attempt + ok)
* Publisher removal (attempt + ok)
* JWT signature verification (attempt + ok)

* warehouse/oidc: add a `verify_for_helper` iface method

This encapsulates the entire JWT verification process. It isn't
hooked up to anything yet, but just to get something down.

* manage/views: add provider names to metrics

* oidc/services: add project tag to metrics during JWT verification

* oidc/services: include provider name in metrics too

* tests/unit: plumb metrics through OIDC unit tests

* tests/unit: fill in coverage

* warehouse: `make translations`

* tests, warehouse: disable `job_workflow_ref`

For now.

* Apply suggestions from code review

Co-authored-by: Dustin Ingram <di@users.noreply.github.com>

* tests, warehouse: update tests for changes

Also use `workflow_filename` consistently.

* warehouse, tests: email all users on OIDC changes

Instead of just owners.

* warehouse, tests: include publisher info in OIDC emails

* warehouse: `make translations`

Co-authored-by: Dustin Ingram <di@users.noreply.github.com>
Co-authored-by: Joachim Jablon <ewjoachim@gmail.com>

dev/environment

@@ -49,3 +49,4 @@ GITHUB_TOKEN_SCANNING_META_API_URL="http://notgithub:8000/meta/public_keys/token
 TWOFACTORREQUIREMENT_ENABLED=true
 TWOFACTORMANDATE_AVAILABLE=true
 TWOFACTORMANDATE_ENABLED=true
+OIDC_ENABLED=true

tests/unit/email/test_init.py

@@ -3575,3 +3575,96 @@ def test_recovery_code_emails(
                 },
             )
         ]
+
+
+class TestOIDCProviderEmails:
+    @pytest.mark.parametrize(
+        "fn, template_name",
+        [
+            (email.send_oidc_provider_added_email, "oidc-provider-added"),
+            (email.send_oidc_provider_removed_email, "oidc-provider-removed"),
+        ],
+    )
+    def test_oidc_provider_emails(
+        self, pyramid_request, pyramid_config, monkeypatch, fn, template_name
+    ):
+        stub_user = pretend.stub(
+            id="id",
+            username="username",
+            name="",
+            email="email@example.com",
+            primary_email=pretend.stub(email="email@example.com", verified=True),
+        )
+        subject_renderer = pyramid_config.testing_add_renderer(
+            f"email/{ template_name }/subject.txt"
+        )
+        subject_renderer.string_response = "Email Subject"
+        body_renderer = pyramid_config.testing_add_renderer(
+            f"email/{ template_name }/body.txt"
+        )
+        body_renderer.string_response = "Email Body"
+        html_renderer = pyramid_config.testing_add_renderer(
+            f"email/{ template_name }/body.html"
+        )
+        html_renderer.string_response = "Email HTML Body"
+
+        send_email = pretend.stub(
+            delay=pretend.call_recorder(lambda *args, **kwargs: None)
+        )
+        pyramid_request.task = pretend.call_recorder(lambda *args, **kwargs: send_email)
+        monkeypatch.setattr(email, "send_email", send_email)
+
+        pyramid_request.db = pretend.stub(
+            query=lambda a: pretend.stub(
+                filter=lambda *a: pretend.stub(
+                    one=lambda: pretend.stub(user_id=stub_user.id)
+                )
+            ),
+        )
+        pyramid_request.user = stub_user
+        pyramid_request.registry.settings = {"mail.sender": "noreply@example.com"}
+
+        project_name = "test_project"
+        fakeprovider = pretend.stub(provider_name="fakeprovider")
+        # NOTE: Can't set __str__ using pretend.stub()
+        monkeypatch.setattr(
+            fakeprovider.__class__, "__str__", lambda s: "fakespecifier"
+        )
+
+        result = fn(
+            pyramid_request, stub_user, project_name=project_name, provider=fakeprovider
+        )
+
+        assert result == {
+            "username": stub_user.username,
+            "project_name": project_name,
+            "provider_name": "fakeprovider",
+            "provider_spec": "fakespecifier",
+        }
+        subject_renderer.assert_()
+        body_renderer.assert_(username=stub_user.username, project_name=project_name)
+        html_renderer.assert_(username=stub_user.username, project_name=project_name)
+        assert pyramid_request.task.calls == [pretend.call(send_email)]
+        assert send_email.delay.calls == [
+            pretend.call(
+                f"{stub_user.username} <{stub_user.email}>",
+                {
+                    "subject": "Email Subject",
+                    "body_text": "Email Body",
+                    "body_html": (
+                        "<html>\n<head></head>\n"
+                        "<body><p>Email HTML Body</p></body>\n</html>\n"
+                    ),
+                },
+                {
+                    "tag": "account:email:sent",
+                    "user_id": stub_user.id,
+                    "additional": {
+                        "from_": "noreply@example.com",
+                        "to": stub_user.email,
+                        "subject": "Email Subject",
+                        "redact_ip": False,
+                    },
+                },
+            )
+        ]

tests/unit/manage/test_init.py

@@ -94,13 +94,37 @@ def view(context, request):
         assert request.session.needs_reauthentication.calls == needs_reauth_calls
 
 
-def test_includeme():
+def test_includeme(monkeypatch):
+    settings = {
+        "warehouse.manage.oidc.user_registration_ratelimit_string": "10 per day",
+        "warehouse.manage.oidc.ip_registration_ratelimit_string": "100 per day",
+    }
+
     config = pretend.stub(
         add_view_deriver=pretend.call_recorder(lambda f, over, under: None),
+        register_service_factory=pretend.call_recorder(lambda s, i, **kw: None),
+        registry=pretend.stub(
+            settings=pretend.stub(get=pretend.call_recorder(lambda k: settings.get(k)))
+        ),
     )
 
+    rate_limit_class = pretend.call_recorder(lambda s: s)
+    rate_limit_iface = pretend.stub()
+    monkeypatch.setattr(manage, "RateLimit", rate_limit_class)
+    monkeypatch.setattr(manage, "IRateLimiter", rate_limit_iface)
+
     manage.includeme(config)
 
     assert config.add_view_deriver.calls == [
         pretend.call(manage.reauth_view, over="rendered_view", under="decorated_view")
     ]
+    assert config.register_service_factory.calls == [
+        pretend.call(
+            "10 per day", rate_limit_iface, name="user_oidc.provider.register"
+        ),
+        pretend.call("100 per day", rate_limit_iface, name="ip_oidc.provider.register"),
+    ]
+    assert config.registry.settings.get.calls == [
+        pretend.call("warehouse.manage.oidc.user_registration_ratelimit_string"),
+        pretend.call("warehouse.manage.oidc.ip_registration_ratelimit_string"),
+    ]