Features of SQS

In the above diagram, only the S1, S3, and S5 servers are queried during short polling. Due to this, the message “d” is not delivered to the consumer.

Short polling is useful in applications that have low latency requirements and send frequent polling requests.

Long polling

In long polling, all available servers are queried for messages, and a response is sent only after SQS finds at least one message to deliver to the consumer who made the fetch request, and an empty response is sent only in case our request times out.

By using long polling, we can reduce the number of empty responses sent by SQS. We can also reduce the false empty responses by querying all servers rather than a subset of servers

Since we are charged on every polling request made to the SQS queue, long polling is preferred to reduce cost by reducing the number of empty responses received. However, it is not beneficial to use long polling in case our application expects an immediate response from the SQS queue. In this case, short polling will be prefered.

Visibility timeout

Since Amazon SQS is a distributed service, there is no guarantee that a consumer has received a message from a queue or not. To ensure unprocessed messages are not deleted from the queue, SQS automatically doesn’t delete any message inside a queue. Due to this reason, it’s the responsibility of the user to delete messages after receiving and processing them.

Since messages inside a queue are not deleted immediately after they are fetched by a consumer, there is a high chance that a single message can be processed by more than one consumer. To overcome this problem, SQS uses visibility timeout, a period during which other consumers cannot receive and process a message after it has been sent to one user.

The following slides demonstrate how visibility timeout works.

By default, the visibility timeout is set to 30 seconds. The minimum value we can provide for this attribute is 0 seconds while the maximum is 12 hours. This time period begins as soon as a message is sent to a consumer. During this period, the consumer must process and delete the message from the SQS queue. If a message is not deleted within this period, it becomes available to other consumers.

We can set the visibility timeout attribute for the entire queue or a single message, depending on the time our application takes to process a message. If we are unsure how long our application takes to process a message, we can create a heartbeat for our consumers. In a heartbeat, an estimate for the visibility timeout is provided, and a request to increase this timeout is sent in case our consumer cannot process the message within the given time.

We can update the visibility timeout attribute to increase or decrease the visibility timeout of a message after receiving a message if we need more time to process the message.

Dead-letter queues

At times, messages in a queue can’t be processed by consumers. This can be due to a corrupted payload, an unexpected stage change, or any unhandled conditions in the consumer or the producer application. In these scenarios, storing these unprocessed messages in our SQS queue can be bothersome and costly as they’ll be sent to consumers whenever a polling request is made.

Amazon SQS allows us to create a dead-letter queue, where we can send all unconsumed messages. Through this, we can isolate these messages and try to determine why their processing failed. We can also create CloudWatch alarms, configure them for when messages are sent to a dead-letter queue, and examine their logs to debug why these messages weren’t consumed.

Redrive policies

While creating a dead-letter queue, a redrive policy is used to define the following:

• The SQS queue that we want to use as the dead-letter queue.

• The source queues that will send their unconsumed messages to the dead-letter queue.

• The maxReceiveCount, which is the maximum number of times a message can be polled. A message is transferred to the dead-letter queue if it’s receive count meets the maxReceiveCount.

Key points to remember

Following are some of the things we must keep in mind while creating a dead-letter queue:

• A FIFO queue can only use another FIFO queue as its dead-letter queue. Similarly, we must create another standard queue as its dead-letter queue in the case of standard queues.

• The dead-letter queues and source queues must be created in the same region and AWS account.

• If we set the maxReceiveCount to 1, message failure due to connectivity issues, or client-side errors will also be sent to the dead-letter queues. Due to this, we must ensure we don’t set the maxReceiveCount to a low value.