Support SubConn idleness when connections are lost

[dfawley]

C and Java both have this behavior. Go does not, and it can cause problems where idle clients are actively reconnecting to backends they don't need to use. Note that if a round robin LB policy is active, it will still maintain persistent connections to all backends, which is by design. Pick first, however, will stay disconnected until the next RPC.

[dfawley]

While related, #1719 and #1786 are separate issues.

[menghanl]

This includes when a GOAWAY is received. The client should stay IDLE until a new RPC is started.

[Jun10ng]

I jump this issue from #4282
from my understanding, after client receive GOAWAY, the connection that received GOAWAYshould become IDLE instead of STOP to prevent client creates a new connection and GracefulStop() closes two connection at the same time？

PLMK if I understand it incorrectly.

Thanks!

[menghanl]

No, the old connection that received GOAWAY is DRAINING. The new connection should be in IDLE.

The current behavior is, the new connection will be CONNECTING -> READY. So a new TCP connection is always created immediately, even if there's no RPC to use it.
This feature request is to delay the new connection, until there's a new RPC.

[ipochi]

Any updates on this feature ? @dfawley

In our setup I have a gut feeling this is whats causing the memory leak.

client is Kubernetes api-server and the grpc-server runs as a proxy(apiserver-network-proxy) on the cluster as a pod, acting as a bridge between control plane and worker nodes.

We have a situation where we are seeing memory leaks but not able to identify the root cause of the issue. For some reason setting MaxConnectionIdle doesn't work. However setting MaxConnectionAge and MaxConnectionAgeGrace works but is not ideal as it terminates the connection regardless of activity from the client. Despite setting the mentioned fields, even if there are no active gRPC connections or open filedescriptors as seen in the prometheus metrics, only ~10% memory reduction is seen ?

Could it possibly be the case where we are hitting this issue ?

[dfawley]

Any updates on this feature?

This was implemented in #4613. gRPC will now not reconnect until an RPC is attempted if a connection is lost.

It's unlikely that this kind of thing would fix memory leak issues, although it will make gRPC consume fewer resources when a ClientConn goes idle and the server closes the connection (e.g. due to max idle or max age settings).

Note that MaxConnectionAgeGrace on the server mostly addresses your concerns about "terminates the connection regardless of activity from the client". It can lead to RPCs being hard-stopped after the grace period, but only if they take longer than the grace period. New RPCs will go on a new connection, and the old connection will only be closed when there are no more RPCs on it or when the grace period expires.

[dfawley]

Closing issue since the feature it describes is implemented by #4613.