kv/kvserver: TestDiscoverIntentAcrossLeaseTransferAwayAndBack failed

[cockroach-teamcity]

kv/kvserver.TestDiscoverIntentAcrossLeaseTransferAwayAndBack failed on release-24.1 @ 41e76d2515efb83cd76c7645f1692ac516ebd4b8:

Fatal error:

panic: test timed out after 1h16m37s
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func1({0xc004082008?, 0x0?})
running tests:
	TestDiscoverIntentAcrossLeaseTransferAwayAndBack (1h16m13s)

Stack:

goroutine 560416 [running]:
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:285 +0x57
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x263
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x265
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x45

Log preceding fatal error

github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*jobMonitor).start.func1({0xe8e8918, 0xc007ed9b90})
	github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/pkg/sql/sqlstats/persistedsqlstats/scheduled_job_monitor.go:97 +0x4b5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 14256
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:471 +0x69d

goroutine 15015 [semacquire, 76 minutes]:
sync.runtime_Semacquire(0xc008e6d290?)
	GOROOT/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc008e6d288)
	GOROOT/src/sync/waitgroup.go:116 +0xa5
golang.org/x/sync/errgroup.(*Group).Wait(0xc008e6d280)
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x38
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc008e6d280?, {0xe8e88e0?, 0xc00b98a230?}})
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:139 +0x65
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0xe8e8918, 0xc007fdf260}, {0xc0029adae8, 0x2, 0x9fa5700?})
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:187 +0x199
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc002950f00, {0xe8e8918, 0xc007fdf260}, {0xe9417e0, 0xc007ed9da0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:356 +0xf71
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).start.func2({0xe8e8918, 0xc001e08f90})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:248 +0x5a9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x263
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 14256
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:471 +0x69d

goroutine 16058 [select, 76 minutes]:
github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1()
	github.com/cockroachdb/cockroach/pkg/util/hlc/pkg/util/hlc/hlc.go:270 +0x20c
created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps in goroutine 14256
	github.com/cockroachdb/cockroach/pkg/util/hlc/pkg/util/hlc/hlc.go:262 +0x237

goroutine 17551 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc005e2e008)
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go:1669 +0x23c
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 17544
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go:387 +0x303a

goroutine 25315 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00c135e50, 0x1)
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/controlbuf.go:418 +0x1af
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002780460)
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/controlbuf.go:552 +0x15b
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_server.go:336 +0x216
created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 25283
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_server.go:333 +0x31eb

goroutine 17192 [chan receive, 76 minutes]:

Parameters:

• attempt=1
• race=true
• run=2
• shard=17

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-46175

[stevendanna]

Duplicate of #122102

[arulajmani]

This test is flaky and has since been skipped on master. See #122629, #122102). I'll backport the skip PR to 24.1.

[arulajmani]

#138737.