ReadException "too many concurrent streams" #1987
Comments
|
@visox I'd like to look into this, if it's still relevant. Does the problem still occur? |
|
@kciesielski I stumbled on this as well. |
|
@kciesielski & @visox a workaround for me was just to retry: sttpClient.send(httpRequest).retry(Schedule.fibonacci(1.second).whileOutput(60.seconds.>))I'm guessing in a burst of too many requests this may happen, so eventually it'll succeed if we retry enough. |
|
I encountered a similar issue with an app frequently canceling requests and I think I was able to reproduce it. Here is an example code that can be run using scala-cli: //> using jvm temurin:21
//> using javaOpt -XX:ActiveProcessorCount=4
//> using scala 3.3.4
//> using dep "dev.zio::zio:2.1.14"
//> using dep "com.softwaremill.sttp.client3::core:3.10.2"
//> using dep "com.softwaremill.sttp.client3::zio:3.10.2"
//> using dep "com.softwaremill.sttp.tapir::tapir-core:1.11.13"
//> using dep "com.softwaremill.sttp.tapir::tapir-armeria-server:1.11.13"
import com.linecorp.armeria.common.SessionProtocol
import com.linecorp.armeria.server.Server as ArmeriaServer
import scala.concurrent.Future
import scala.jdk.FutureConverters.*
import sttp.client3.*
import sttp.client3.httpclient.zio.HttpClientZioBackend
import sttp.model.*
import sttp.tapir.*
import sttp.tapir.server.armeria.ArmeriaFutureServerInterpreter
import zio.*
val port = 8080
def randomPause =
Random.nextIntBounded(100).flatMap(n => ZIO.sleep(Duration.fromNanos(n)))
def runClient =
val request = basicRequest.get(uri"http://localhost:$port/test").response(asString)
for
backend <- HttpClientZioBackend.scoped()
_ <- request.send(backend) // initialize the connection
requests = List.fill(20) {
randomPause *>
request.send(backend).catchAll(e => ZIO.succeed(e.printStackTrace())) *> // expecting an error here
randomPause
}
_ <- {
ZIO.log("Sending requests") *>
ZIO.raceAll(requests.head, requests.tail)
}.repeat(Schedule.spaced(Duration.fromMillis(500)))
yield ()
def startServer =
val responseText = "Test".repeat(100)
for
runtime <- ZIO.runtime[Any]
_ <- ZIO.acquireRelease {
ZIO.fromFuture { implicit ec =>
val testEndpoint =
endpoint.get.out(stringBody).serverLogic { _ =>
Unsafe.unsafe[Future[Either[Unit, String]]] { implicit unsafe =>
runtime.unsafe.runToFuture(randomPause.as(Right(responseText)))
}
}
val tapirService = ArmeriaFutureServerInterpreter().toService(testEndpoint)
val server = ArmeriaServer
.builder()
.port(port, SessionProtocol.HTTP)
.http2MaxStreamsPerConnection(50)
.http2MaxResetFramesPerWindow(Int.MaxValue, 1)
.service(tapirService)
.build()
server.start().asScala.map(_ => server)
}
} { server =>
ZIO.fromFuture(_ => server.stop().asScala).orDie
}
yield ()
object Demo extends ZIOAppDefault:
override def run =
for
_ <- startServer
_ <- runClient
yield ()This code repeatedly creates 20 fibers that send requests, and cancels them as soon as one of them completes. On my PC it starts logging "too many concurrent streams" errors after 1-2 minutes, even though the number of concurrent requests is always below the limit. Using the debugger, I found that the The documentation for HttpClient says:
But it seems the |
|
@ioaoue thanks for a reproducer! I confirm that it happens on my machine as well. Not sure about the interruption point you mentioned, though. The |
|
If I understand correctly, the
In case of |
|
Ah, yes you're right of course. Now we just need to come up with a proper fix :) |
|
See PR #2413 for a PoC of a fix. The code you posted now runs for a long time without throwing exceptions. We can use ZIO's acquire/release to properly manage the lifecycle of the publisher. This has several consequences:
|
|
Actually, it's a bit better - I forgot (but got reminded ;) ) that the futures returned by HttpClient are in fact cancellable (despite the javadoc saying something else). The only problem is that cancellation is asynchronous, which is not perfect, but as good as it gets (we can't obtain information if cancellation is done - same problem as in #1746). So we can use |
Hi i am running a service which makes lots of calls to some other service, lets say 10 calls per second.
The service runs properly but eventually there are periods when it starts to crash with
Also its a POST request with some timeout a json body and some auth headers
in sttp code i do see
and also from the stack trace one can see its an IO exception with
too many concurrent streamsAny obvious solution ?
The text was updated successfully, but these errors were encountered: