Merge pull request #3103 from redpanda-data/sqs

aws/sqs: prevent ackLoop starvation

CHANGELOG.md

@@ -7,6 +7,7 @@ All notable changes to this project will be documented in this file.
 
 ### Added
 
+- `aws_sqs` now has a `max_outstanding` field to prevent unbounded memory usage. (@rockwotj)
 - `avro` scanner now emits metadata for the Avro schema it used along with the schema fingerprint. (@rockwotj)
 - Field `content_type` added to the `amqp_1` output. (@timo102)
 - `kafka_franz`, `ockam_kafka`, `redpanda`, `redpanda_common`, `redpanda_migrator` now support `fetch_max_wait` configuration field.
@@ -18,6 +19,7 @@ All notable changes to this project will be documented in this file.
 
 - The `code` and `file` fields on the `javascript` processor docs no longer erroneously mention interpolation support. (@mihaitodor)
 - The `postgres_cdc` now correctly handles `null` values. (@rockwotj)
+- Fix an issue in `aws_sqs` with refreshing in-flight message leases which could prevent acks from processed. (@rockwotj)
 
 ## 4.44.0 - 2024-12-13
 

docs/modules/components/pages/inputs/aws_sqs.adoc

@@ -38,6 +38,7 @@ input:
   label: ""
   aws_sqs:
     url: "" # No default (required)
+    max_outstanding_messages: 1000
 ```
 
 --
@@ -54,7 +55,9 @@ input:
     delete_message: true
     reset_visibility: true
     max_number_of_messages: 10
+    max_outstanding_messages: 1000
     wait_time_seconds: 0
+    message_timeout: 30s
     region: ""
     endpoint: ""
     credentials:
@@ -127,6 +130,15 @@ The maximum number of messages to return on one poll. Valid values: 1 to 10.
 
 *Default*: `10`
 
+=== `max_outstanding_messages`
+
+The maximum number of outstanding pending messages to be consumed at a given time.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
 === `wait_time_seconds`
 
 Whether to set the wait time. Enabling this activates long-polling. Valid values: 0 to 20.
@@ -136,6 +148,15 @@ Whether to set the wait time. Enabling this activates long-polling. Valid values
 
 *Default*: `0`
 
+=== `message_timeout`
+
+The time to process messages before needing to refresh the receipt handle. Messages will be eligible for refresh when half of the timeout has elapsed. This sets MessageVisibility for each received message.
+
+
+*Type*: `string`
+
+*Default*: `"30s"`
+
 === `region`
 
 The AWS region to target.