bug: Intermittent failure in TestDomainRegisterFail

[peterbroadhurst]

What happened?

Intermittent failure in TestDomainRegisterFail.

Example build: https://github.com/LF-Decentralized-Trust-labs/paladin/actions/runs/12948362537/job/36116875354

2025-01-24T11:11:14.1940100Z === RUN   TestDomainRegisterFail
2025-01-24T11:11:14.1940862Z [2025-01-24T11:09:43.929Z]  INFO server starting on unix:/tmp/ut_2702291982.sock
2025-01-24T11:11:14.1941650Z [2025-01-24T11:09:43.929Z]  INFO server listening at /tmp/ut_2702291982.sock
2025-01-24T11:11:14.1942393Z [2025-01-24T11:09:43.929Z]  INFO Run GRPC Server
2025-01-24T11:11:14.1942919Z [2025-01-24T11:09:43.929Z]  INFO Server started
2025-01-24T11:11:14.1943572Z [2025-01-24T11:09:43.931Z]  INFO Plugin loader connected plugin_reqid=4jXQS4X8
2025-01-24T11:11:14.1944302Z [2025-01-24T11:09:43.931Z] DEBUG 0 of 1 DOMAIN plugins loaded
2025-01-24T11:11:14.1945117Z [2025-01-24T11:09:43.931Z]  INFO All TRANSPORT plugins loaded []
2025-01-24T11:11:14.1945796Z [2025-01-24T11:09:43.931Z]  INFO All REGISTRY plugins loaded []
2025-01-24T11:11:14.1946445Z [2025-01-24T11:09:43.931Z] DEBUG 0 of 1 DOMAIN plugins loaded
2025-01-24T11:11:14.1947098Z [2025-01-24T11:09:43.931Z]  INFO All TRANSPORT plugins loaded []
2025-01-24T11:11:14.1947772Z [2025-01-24T11:09:43.931Z]  INFO All REGISTRY plugins loaded []
2025-01-24T11:11:14.1961091Z [2025-01-24T11:09:43.932Z] ERROR Plugin factory failed d921fb0c-651f-4e43-80a7-7c741e7f34a6: pop plugin=DOMAIN[domain1/d921fb0c-651f-4e43-80a7-7c741e7f34a6]
2025-01-24T11:11:14.1962888Z [2025-01-24T11:09:43.932Z]  INFO Plugin stopped id:"d921fb0c-651f-4e43-80a7-7c741e7f34a6" name:"domain1" [d921fb0c-651f-4e43-80a7-7c741e7f34a6] (type=DOMAIN)
2025-01-24T11:11:14.1964298Z [2025-01-24T11:09:43.933Z] DEBUG domain handler ending plugin=DOMAIN[domain1/d921fb0c-651f-4e43-80a7-7c741e7f34a6]
2025-01-24T11:11:14.1965269Z coverage: 99.0% of statements
2025-01-24T11:11:14.1965700Z panic: test timed out after 1m30s
2025-01-24T11:11:14.1966127Z running tests:
2025-01-24T11:11:14.1966507Z 	TestDomainRegisterFail (1m30s)
2025-01-24T11:11:14.1966793Z 
2025-01-24T11:11:14.1966946Z goroutine 87 [running]:
2025-01-24T11:11:14.1967344Z testing.(*M).startAlarm.func1()
2025-01-24T11:11:14.1968031Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/testing/testing.go:2366 +0x385
2025-01-24T11:11:14.1968680Z created by time.goFunc
2025-01-24T11:11:14.1969270Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/time/sleep.go:177 +0x2d
2025-01-24T11:11:14.1969719Z 
2025-01-24T11:11:14.1969895Z goroutine 1 [chan receive]:
2025-01-24T11:11:14.1970415Z testing.(*T).Run(0xc0000f2b60, {0x116f4aa?, 0x0?}, 0x12a9748)
2025-01-24T11:11:14.1971187Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/testing/testing.go:1750 +0x3ab
2025-01-24T11:11:14.1971851Z testing.runTests.func1(0xc0000f2b60)
2025-01-24T11:11:14.1972533Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/testing/testing.go:2161 +0x37
2025-01-24T11:11:14.1973199Z testing.tRunner(0xc0000f2b60, 0xc00052dc60)
2025-01-24T11:11:14.1973887Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/testing/testing.go:1689 +0xfb
2025-01-24T11:11:14.1974912Z testing.runTests(0xc0003a2300, {0x1b183c0, 0x1c, 0x1c}, {0x1?, 0x527b4e?, 0x1b990c0?})
2025-01-24T11:11:14.1975837Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/testing/testing.go:2159 +0x445
2025-01-24T11:11:14.1976471Z testing.(*M).Run(0xc00022eaa0)
2025-01-24T11:11:14.1977105Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/testing/testing.go:2027 +0x68b
2025-01-24T11:11:14.1977698Z main.main()
2025-01-24T11:11:14.1978059Z 	_testmain.go:135 +0x193
2025-01-24T11:11:14.1978310Z 
2025-01-24T11:11:14.1978461Z goroutine 63 [select]:
2025-01-24T11:11:14.1979105Z google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000531d80, 0x1)
2025-01-24T11:11:14.1980212Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/controlbuf.go:412 +0x108
2025-01-24T11:11:14.1981213Z google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000649380)
2025-01-24T11:11:14.1982247Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/controlbuf.go:575 +0x86
2025-01-24T11:11:14.1983182Z google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
2025-01-24T11:11:14.1984190Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/http2_client.go:470 +0xd2
2025-01-24T11:11:14.2076978Z created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 60
2025-01-24T11:11:14.2078377Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/http2_client.go:468 +0x2472
2025-01-24T11:11:14.2079165Z 
2025-01-24T11:11:14.2079346Z goroutine 55 [chan receive]:
2025-01-24T11:11:14.2080149Z github.com/kaleido-io/paladin/core/internal/plugins.TestDomainRegisterFail(0xc0000f24e0)
2025-01-24T11:11:14.2081382Z 	/home/runner/work/paladin/paladin/core/go/internal/plugins/domains_test.go:457 +0x1e6
2025-01-24T11:11:14.2082234Z testing.tRunner(0xc0000f24e0, 0x12a9748)
2025-01-24T11:11:14.2082988Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/testing/testing.go:1689 +0xfb
2025-01-24T11:11:14.2083716Z created by testing.(*T).Run in goroutine 1
2025-01-24T11:11:14.2084461Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/testing/testing.go:1742 +0x390
2025-01-24T11:11:14.2085114Z 
2025-01-24T11:11:14.2085284Z goroutine 56 [IO wait]:
2025-01-24T11:11:14.2085786Z internal/poll.runtime_pollWait(0x7fa7396c63e8, 0x72)
2025-01-24T11:11:14.2086601Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/runtime/netpoll.go:345 +0x85
2025-01-24T11:11:14.2087326Z internal/poll.(*pollDesc).wait(0x3?, 0x10?, 0x0)
2025-01-24T11:11:14.2088367Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
2025-01-24T11:11:14.2089102Z internal/poll.(*pollDesc).waitRead(...)
2025-01-24T11:11:14.2089826Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/internal/poll/fd_poll_runtime.go:89
2025-01-24T11:11:14.2090524Z internal/poll.(*FD).Accept(0xc0006b1000)
2025-01-24T11:11:14.2091245Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/internal/poll/fd_unix.go:611 +0x2ac
2025-01-24T11:11:14.2091924Z net.(*netFD).accept(0xc0006b1000)
2025-01-24T11:11:14.2092561Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/net/fd_unix.go:172 +0x29
2025-01-24T11:11:14.2093190Z net.(*UnixListener).accept(0x44e580?)
2025-01-24T11:11:14.2093880Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/net/unixsock_posix.go:172 +0x16
2025-01-24T11:11:14.2094807Z net.(*UnixListener).Accept(0xc0006abd10)
2025-01-24T11:11:14.2095510Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/net/unixsock.go:260 +0x30
2025-01-24T11:11:14.2096387Z google.golang.org/grpc.(*Server).Serve(0xc000595000, {0x13c5cd0, 0xc0006abd10})
2025-01-24T11:11:14.2097323Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:884 +0x49e
2025-01-24T11:11:14.2098483Z github.com/kaleido-io/paladin/core/internal/plugins.(*pluginManager).runServer(0xc00018a2d0, {0x13c82a0, 0x1bfbbc0})
2025-01-24T11:11:14.2099794Z 	/home/runner/work/paladin/paladin/core/go/internal/plugins/plugin_manager.go:173 +0xde
2025-01-24T11:11:14.2100967Z created by github.com/kaleido-io/paladin/core/internal/plugins.(*pluginManager).Start in goroutine 55
2025-01-24T11:11:14.2102187Z 	/home/runner/work/paladin/paladin/core/go/internal/plugins/plugin_manager.go:165 +0x34f
2025-01-24T11:11:14.2102812Z 
2025-01-24T11:11:14.2102969Z goroutine 57 [select]:
2025-01-24T11:11:14.2103783Z google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0001dde10, {0x13c83b8, 0xc0006a70e0})
2025-01-24T11:11:14.2115354Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/grpcsync/callback_serializer.go:88 +0x115
2025-01-24T11:11:14.2116547Z created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 55
2025-01-24T11:11:14.2117787Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/grpcsync/callback_serializer.go:52 +0x11a
2025-01-24T11:11:14.2118479Z 
2025-01-24T11:11:14.2118630Z goroutine 58 [select]:
2025-01-24T11:11:14.2119447Z google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0001dde80, {0x13c83b8, 0xc0006a7130})
2025-01-24T11:11:14.2120774Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/grpcsync/callback_serializer.go:88 +0x115
2025-01-24T11:11:14.2121967Z created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 55
2025-01-24T11:11:14.2123203Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/grpcsync/callback_serializer.go:52 +0x11a
2025-01-24T11:11:14.2123896Z 
2025-01-24T11:11:14.2124173Z goroutine 62 [IO wait]:
2025-01-24T11:11:14.2124799Z internal/poll.runtime_pollWait(0x7fa7396c62f0, 0x72)
2025-01-24T11:11:14.2125579Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/runtime/netpoll.go:345 +0x85
2025-01-24T11:11:14.2126467Z internal/poll.(*pollDesc).wait(0xc0006b1500?, 0xc0006c4000?, 0x0)
2025-01-24T11:11:14.2127352Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
2025-01-24T11:11:14.2128081Z internal/poll.(*pollDesc).waitRead(...)
2025-01-24T11:11:14.2128808Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/internal/poll/fd_poll_runtime.go:89
2025-01-24T11:11:14.2129634Z internal/poll.(*FD).Read(0xc0006b1500, {0xc0006c4000, 0x8000, 0x8000})
2025-01-24T11:11:14.2130475Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/internal/poll/fd_unix.go:164 +0x27a
2025-01-24T11:11:14.2131286Z net.(*netFD).Read(0xc0006b1500, {0xc0006c4000?, 0x1060100000000?, 0x8?})
2025-01-24T11:11:14.2132069Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/net/fd_posix.go:55 +0x25
2025-01-24T11:11:14.2132795Z net.(*conn).Read(0xc0003e63a8, {0xc0006c4000?, 0x800010601?, 0x0?})
2025-01-24T11:11:14.2133623Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/net/net.go:185 +0x45
2025-01-24T11:11:14.2134422Z bufio.(*Reader).Read(0xc0006a89c0, {0xc0001b5540, 0x9, 0xc000500008?})
2025-01-24T11:11:14.2135350Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/bufio/bufio.go:241 +0x197
2025-01-24T11:11:14.2136112Z io.ReadAtLeast({0x13bba00, 0xc0006a89c0}, {0xc0001b5540, 0x9, 0x9}, 0x9)
2025-01-24T11:11:14.2136885Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/io/io.go:335 +0x90
2025-01-24T11:11:14.2137425Z io.ReadFull(...)
2025-01-24T11:11:14.2137889Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/io/io.go:354
2025-01-24T11:11:14.2138786Z golang.org/x/net/http2.readFrameHeader({0xc0001b5540, 0x9, 0xc0000487e0?}, {0x13bba00?, 0xc0006a89c0?})
2025-01-24T11:11:14.2139838Z 	/home/runner/go/pkg/mod/golang.org/x/net@v0.33.0/http2/frame.go:237 +0x65
2025-01-24T11:11:14.2140600Z golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001b5500)
2025-01-24T11:11:14.2141391Z 	/home/runner/go/pkg/mod/golang.org/x/net@v0.33.0/http2/frame.go:501 +0x85
2025-01-24T11:11:14.2142377Z google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0001bad88, 0xc0006a8a20)
2025-01-24T11:11:14.2143577Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/http2_client.go:1645 +0x1d0
2025-01-24T11:11:14.2144812Z created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 60
2025-01-24T11:11:14.2145954Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/http2_client.go:412 +0x1e79
2025-01-24T11:11:14.2146608Z 
2025-01-24T11:11:14.2146763Z goroutine 65 [select]:
2025-01-24T11:11:14.2147654Z google.golang.org/grpc/internal/transport.(*recvBufferReader).readHeaderClient(0xc0006a76d0, {0xc0006c0790, 0x5, 0x5})
2025-01-24T11:11:14.2148991Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/transport.go:199 +0xa7
2025-01-24T11:11:14.2150393Z google.golang.org/grpc/internal/transport.(*recvBufferReader).ReadHeader(0xc0006a76d0, {0xc0006c0790?, 0xc0001bad88?, 0xc0006b6d80?})
2025-01-24T11:11:14.2152109Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/transport.go:145 +0x45
2025-01-24T11:11:14.2153455Z google.golang.org/grpc/internal/transport.(*transportReader).ReadHeader(0xc0003833c0, {0xc0006c0790?, 0xc0003ae195?, 0x46?})
2025-01-24T11:11:14.2155045Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/transport.go:614 +0x25
2025-01-24T11:11:14.2156214Z google.golang.org/grpc/internal/transport.(*Stream).ReadHeader(0xc0006b6d80, {0xc0006c0790, 0x5, 0x5})
2025-01-24T11:11:14.2157417Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/transport.go:557 +0xa6
2025-01-24T11:11:14.2158355Z google.golang.org/grpc.(*parser).recvMsg(0xc0006c0780, 0x400000)
2025-01-24T11:11:14.2159209Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/rpc_util.go:659 +0x3e
2025-01-24T11:11:14.2160435Z google.golang.org/grpc.recvAndDecompress(0xc0006c0780, 0xc0006b6d80, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0}, 0x0)
2025-01-24T11:11:14.2161564Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/rpc_util.go:823 +0x9a
2025-01-24T11:11:14.2162781Z google.golang.org/grpc.recv(0x0?, {0x7fa7373ad168, 0x1bfbbc0}, 0x115a2d7?, {0x0?, 0x0?}, {0x10dd260, 0xc0006a8c00}, 0x8?, 0x0, ...)
2025-01-24T11:11:14.2163921Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/rpc_util.go:916 +0x91
2025-01-24T11:11:14.2165054Z google.golang.org/grpc.(*csAttempt).recvMsg(0xc0003465b0, {0x10dd260, 0xc0006a8c00}, 0xc0006b6b40?)
2025-01-24T11:11:14.2166084Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/stream.go:1129 +0x2ee
2025-01-24T11:11:14.2166865Z google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x0?)
2025-01-24T11:11:14.2167679Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/stream.go:969 +0x1f
2025-01-24T11:11:14.2168617Z google.golang.org/grpc.(*clientStream).withRetry(0xc0006b6b40, 0xc000095e78, 0xc000095e68)
2025-01-24T11:11:14.2169610Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/stream.go:773 +0x3ae
2025-01-24T11:11:14.2170643Z google.golang.org/grpc.(*clientStream).RecvMsg(0xc0006b6b40, {0x10dd260?, 0xc0006a8c00?})
2025-01-24T11:11:14.2171909Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/stream.go:968 +0x17f
2025-01-24T11:11:14.2172729Z google.golang.org/grpc.(*GenericClientStream[...]).Recv(0x13d03e0)
2025-01-24T11:11:14.2173815Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/stream_interfaces.go:185 +0x4e
2025-01-24T11:11:14.2174999Z github.com/kaleido-io/paladin/core/internal/plugins.(*testPluginLoader).Run(0xc0002a6930)
2025-01-24T11:11:14.2176191Z 	/home/runner/work/paladin/paladin/core/go/internal/plugins/unittest_plugin_loader.go:82 +0xfa
2025-01-24T11:11:14.2177341Z github.com/kaleido-io/paladin/core/internal/plugins.newTestDomainPluginManager.func1()
2025-01-24T11:11:14.2178456Z 	/home/runner/work/paladin/paladin/core/go/internal/plugins/domains_test.go:105 +0x52
2025-01-24T11:11:14.2179646Z created by github.com/kaleido-io/paladin/core/internal/plugins.newTestDomainPluginManager in goroutine 55
2025-01-24T11:11:14.2180858Z 	/home/runner/work/paladin/paladin/core/go/internal/plugins/domains_test.go:103 +0x472
2025-01-24T11:11:14.2181461Z 
2025-01-24T11:11:14.2181620Z goroutine 59 [select]:
2025-01-24T11:11:14.2182432Z google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0001ddee0, {0x13c83b8, 0xc0006a7180})
2025-01-24T11:11:14.2183742Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/grpcsync/callback_serializer.go:88 +0x115
2025-01-24T11:11:14.2185122Z created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 55
2025-01-24T11:11:14.2186338Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/grpcsync/callback_serializer.go:52 +0x11a
2025-01-24T11:11:14.2187003Z 
2025-01-24T11:11:14.2187158Z goroutine 64 [select]:
2025-01-24T11:11:14.2187670Z google.golang.org/grpc.newClientStreamWithParams.func4()
2025-01-24T11:11:14.2188501Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/stream.go:393 +0x8c
2025-01-24T11:11:14.2189379Z created by google.golang.org/grpc.newClientStreamWithParams in goroutine 55
2025-01-24T11:11:14.2190343Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/stream.go:392 +0xe08
2025-01-24T11:11:14.2190852Z 
2025-01-24T11:11:14.2191003Z goroutine 47 [select]:
2025-01-24T11:11:14.2191978Z github.com/kaleido-io/paladin/core/internal/plugins.(*pluginManager).sendPluginsToLoader(0xc00018a2d0, {0x13ce620, 0xc000724180})
2025-01-24T11:11:14.2193362Z 	/home/runner/work/paladin/paladin/core/go/internal/plugins/plugin_manager.go:392 +0x271
2025-01-24T11:11:14.2194910Z github.com/kaleido-io/paladin/core/internal/plugins.(*pluginManager).InitLoader(0xc00018a2d0, 0xc0003dd180, {0x13ce620, 0xc000724180})
2025-01-24T11:11:14.2196348Z 	/home/runner/work/paladin/paladin/core/go/internal/plugins/plugin_manager.go:274 +0x2c9
2025-01-24T11:11:14.2197882Z github.com/kaleido-io/paladin/toolkit/pkg/prototk._PluginController_InitLoader_Handler({0x1121d80, 0xc00018a2d0}, {0x13cc0e0, 0xc0005be2d0})
2025-01-24T11:11:14.2199322Z 	/home/runner/work/paladin/paladin/toolkit/go/pkg/prototk/service_grpc.pb.go:200 +0x107
2025-01-24T11:11:14.2200892Z google.golang.org/grpc.(*Server).processStreamingRPC(0xc000595000, {0x13c8380, 0xc00070c4e0}, {0x13cf8a0, 0xc0003421a0}, 0xc00072a000, 0xc0006abe30, 0x1b13ae0, 0x0)
2025-01-24T11:11:14.2202309Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1688 +0x1208
2025-01-24T11:11:14.2203331Z google.golang.org/grpc.(*Server).handleStream(0xc000595000, {0x13cf8a0, 0xc0003421a0}, 0xc00072a000)
2025-01-24T11:11:14.2204384Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1809 +0xe3a
2025-01-24T11:11:14.2205314Z google.golang.org/grpc.(*Server).serveStreams.func2.1()
2025-01-24T11:11:14.2206132Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1029 +0x8b
2025-01-24T11:11:14.2207024Z created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 46
2025-01-24T11:11:14.2207982Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1040 +0x125
2025-01-24T11:11:14.2208493Z 
2025-01-24T11:11:14.2208828Z goroutine 44 [select]:
2025-01-24T11:11:14.2209496Z google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0003dc140, 0x1)
2025-01-24T11:11:14.2210614Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/controlbuf.go:412 +0x108
2025-01-24T11:11:14.2211644Z google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00070a200)
2025-01-24T11:11:14.2212707Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/controlbuf.go:575 +0x86
2025-01-24T11:11:14.2213703Z google.golang.org/grpc/internal/transport.NewServerTransport.func2()
2025-01-24T11:11:14.2214929Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/http2_server.go:335 +0xde
2025-01-24T11:11:14.2216168Z created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 43
2025-01-24T11:11:14.2217379Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/http2_server.go:333 +0x18fe
2025-01-24T11:11:14.2218051Z 
2025-01-24T11:11:14.2218217Z goroutine 45 [select]:
2025-01-24T11:11:14.2218878Z google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0003421a0)
2025-01-24T11:11:14.2220023Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/http2_server.go:1183 +0x205
2025-01-24T11:11:14.2221162Z created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 43
2025-01-24T11:11:14.2222357Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/http2_server.go:356 +0x1945
2025-01-24T11:11:14.2223018Z 
2025-01-24T11:11:14.2223179Z goroutine 46 [IO wait]:
2025-01-24T11:11:14.2223652Z internal/poll.runtime_pollWait(0x7fa7396c65d8, 0x72)
2025-01-24T11:11:14.2224407Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/runtime/netpoll.go:345 +0x85
2025-01-24T11:11:14.2225332Z internal/poll.(*pollDesc).wait(0xc00070a080?, 0xc000712000?, 0x0)
2025-01-24T11:11:14.2226212Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
2025-01-24T11:11:14.2226942Z internal/poll.(*pollDesc).waitRead(...)
2025-01-24T11:11:14.2227668Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/internal/poll/fd_poll_runtime.go:89
2025-01-24T11:11:14.2228481Z internal/poll.(*FD).Read(0xc00070a080, {0xc000712000, 0x8000, 0x8000})
2025-01-24T11:11:14.2229323Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/internal/poll/fd_unix.go:164 +0x27a
2025-01-24T11:11:14.2230134Z net.(*netFD).Read(0xc00070a080, {0xc000712000?, 0xc0005bdc10?, 0x415cdb?})
2025-01-24T11:11:14.2230933Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/net/fd_posix.go:55 +0x25
2025-01-24T11:11:14.2231668Z net.(*conn).Read(0xc000376150, {0xc000712000?, 0x18?, 0x7fa7803785b8?})
2025-01-24T11:11:14.2232425Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/net/net.go:185 +0x45
2025-01-24T11:11:14.2233125Z bufio.(*Reader).Read(0xc000710000, {0xc000680ba0, 0x9, 0x95c94d?})
2025-01-24T11:11:14.2233998Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/bufio/bufio.go:241 +0x197
2025-01-24T11:11:14.2234908Z io.ReadAtLeast({0x13bba00, 0xc000710000}, {0xc000680ba0, 0x9, 0x9}, 0x9)
2025-01-24T11:11:14.2235784Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/io/io.go:335 +0x90
2025-01-24T11:11:14.2236327Z io.ReadFull(...)
2025-01-24T11:11:14.2236794Z 	/opt/hostedtoolcache/go/1.22.11/x64/src/io/io.go:354
2025-01-24T11:11:14.2237687Z golang.org/x/net/http2.readFrameHeader({0xc000680ba0, 0x9, 0xc000428460?}, {0x13bba00?, 0xc000710000?})
2025-01-24T11:11:14.2238738Z 	/home/runner/go/pkg/mod/golang.org/x/net@v0.33.0/http2/frame.go:237 +0x65
2025-01-24T11:11:14.2239496Z golang.org/x/net/http2.(*Framer).ReadFrame(0xc000680b60)
2025-01-24T11:11:14.2240279Z 	/home/runner/go/pkg/mod/golang.org/x/net@v0.33.0/http2/frame.go:501 +0x85
2025-01-24T11:11:14.2241468Z google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0003421a0, {0x13c8380, 0xc00070c1e0}, 0xc00070c210)
2025-01-24T11:11:14.2242869Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/internal/transport/http2_server.go:644 +0x10d
2025-01-24T11:11:14.2244330Z google.golang.org/grpc.(*Server).serveStreams(0xc000595000, {0x13c82a0?, 0x1bfbbc0?}, {0x13cf8a0, 0xc0003421a0}, {0x13cf180?, 0xc000376150?})
2025-01-24T11:11:14.2245803Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1023 +0x3b6
2025-01-24T11:11:14.2246583Z google.golang.org/grpc.(*Server).handleRawConn.func1()
2025-01-24T11:11:14.2247391Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:958 +0x56
2025-01-24T11:11:14.2248246Z created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 43
2025-01-24T11:11:14.2249147Z 	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:957 +0x1c6
2025-01-24T11:11:14.2249989Z FAIL	github.com/kaleido-io/paladin/core/internal/plugins	90.140s

What did you expect to happen?

Test passed consistently

How can we reproduce it (as minimally and precisely as possible)?

Intermittent test failure

Anything else we need to know?

No response

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here