Add possibility to acquire permits on primary shards with different checks

[arteam]

Since #42241 we check that the shard must be in a primary mode for acquiring a primary permit on it. We would like customize this check and an option to perform different checks before running the onPermitAcquired listener. For example, we would to skip the primary mode check when we acquire primary permits during recovering of a hollow indexing shard.

See ES-10487

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[kingherc]

Looking good. A couple of comments.

[kingherc]

LGTM

[fcofdez]

I'm a bit late, but wouldn't be possible to somehow relax the assertion in #isPrimaryMode when the engine is hollow instead?

[kingherc]

Hi @fcofdez ! It's possible, but it's unclear what's the best way to do it since the hollow info/logic is in serverless code. In the POC, I remember I simply relaxed the assertion by allowing it if the shard is in the recovery process. Not sure though it's better than this PR since it was more generic (recovering shards). Feel free to shoot a specific proposal though to see if it's better.

[arteam]

@fcofdez I believe the reason was that IndexShard doesn't know about HollowEngine which is a serverless concept, so you would need to expose this abstraction to Engine. I think disabling the primary mode check explicitly seems to be a good first approach and we can revert to the engine check if that makes more sense.

[fcofdez]

I've been chatting with @kingherc and I think that we could avoid this change if we acquire the permits in the target node during the primary context handover in serverless as that's called before the shard is marked as started in

elasticsearch/server/src/main/java/org/elasticsearch/indices/cluster/IndicesClusterStateService.java

Lines 974 to 988 in a59c182

	@Override
	public void onRecoveryDone(
	final RecoveryState state,
	ShardLongFieldRange timestampMillisFieldRange,
	ShardLongFieldRange eventIngestedMillisFieldRange
	) {
	shardStateAction.shardStarted(
	shardRouting,
	primaryTerm,
	"after " + state.getRecoverySource(),
	timestampMillisFieldRange,
	eventIngestedMillisFieldRange,
	ActionListener.noop()
	);
	}

. That would avoid having to change this API.

[kingherc]

@fcofdez agreed. Opened https://elasticco.atlassian.net/browse/ES-10537 .