-
A potential issue crossed my mind in my use case for using rqueue as a job queuing mechanism for short lived jobs (Let's say no more than 10 seconds hopefully). From my understanding timeouts work great for issues like a restarted/failed consumer. However there's an edge case I'm concerned with and not sure if queues implementation handles it gracefully or not. What happens when a time out occurs but the job is still healthy (lets just say its taking abnormally long but is completing), does the job get killed on the original consumer before it's retried on another consumer, or do I potentially have two tries of the same job running at the same time? This issue popped in my head after reviewing the question about increasing the timeout via a "heartbeat" call for long running jobs, making me wonder what would happen if the heartbeat wasn't setup properly or didn't happen fast enough essentially, or wasn't used at all. Would the original job attempt keep moving on and then cause rqueue to send the work to another one, creating 2 instances of the same job? Or is there real locking and the first job instance is forcibly killed before being handed to another consumer? I'm pretty new with rqueue so please bear with me, I appreciate any help or information, I just couldn't find enough from the wiki to determine the answer on my own. Rqueue has been pretty robust so far for my needs and appreciate the work done engineering it, I am just trying to determine potential problems. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
Rqueue will process two parallel jobs if current job has timed-out due to slow processing and another listener is still working. So timeout/visibilityTimeout should be configured with some delta like 5 seconds or the worst case execution time of your listener. |
Beta Was this translation helpful? Give feedback.
Rqueue will process two parallel jobs if current job has timed-out due to slow processing and another listener is still working. So timeout/visibilityTimeout should be configured with some delta like 5 seconds or the worst case execution time of your listener.