How to manage a rogue client that is not closing connections

[pkpfr]

We have a public gRPC API. We have a client that is consuming our API based on the REST paradigm of creating a connection (channel) for every request. We suspect that they are not closing this channel once the request has been made.

On the server side, everything functions ok for a while, then is seems that something is exhausted. Requests back up on the servers and are not processed - this results in our proxy timing out and sending an unavailable response. Restarting the server fixes the issue.

Unfortunately, it seems that there is no way to monitor what is happening on the server side and prune these connections. We have the following keep alive settings, but they don't appear to have an impact:

grpc.KeepaliveParams(keepalive.ServerParameters{
		MaxConnectionIdle:     time.Minute * 5,
		MaxConnectionAge:      time.Minute * 15,
		MaxConnectionAgeGrace: time.Minute * 1,
		Time:    time.Second * 60,
		Timeout: time.Second * 10,
	})

Is there any way that we can monitor channel creation and destruction on the server side - if only to prove to the client that their consumption is causing the problems. Verbose logging has not been helpful as it seems to only log the client activity on the server (I.e. the server consuming pub/sub and logging as a client). I have also looked a channelz, but we have mutual TLS auth and I have been unsuccessful in being able to get it to work on our production pods.

We have instructed our client to use a single channel, and if that is not possible, to close the channels that they are creating, but they are a corporate and move very slowly.

[menghanl]

Is there any way that we can monitor channel creation and destruction on the server side

Servers don't see channels. They only see the TCP connections created by the clients. I think you can count the number of connections from each client (by IP).
In the normal case, each channel should only need to create one connection to one backend (unless you intentionally create a TCP pool). If you are getting multiple connections from each client IP, but are only expecting 1, that kinds of proves the problem.

To count number of connections, you can do it by wrapping the listerner and overriding Accept(). Or use the stats handler.

to use a single channel, and if that is not possible, to close the channels that they are creating

You are essentially DoS'ed by your clients. So this would be the right thing to do.
You can try to limit the number of connection each server accepts. But the hard part is, since you cannot tell which client connection are from the stale channels, you won't know which to reject.

[stale]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

[fm0803]

I face a similar problem. I set keepalive policy as above, expecting to close a client that does nothing but does not close connection, however, after 2 hours, this client stays alive and can send requests normally.

[stale]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

[menghanl]

Another thing to mention is, go's implementation is missing the IDLE feature: #1719, #1786

With IDLE, the unused clients would drop the connections. We will try to prioritize this feature.

[dfawley]

Let's track this part of the feature under #4298.