Fanout Architecture using Amazon SNS and SQS

Learn about the fanout architecture and how to deploy it using Amazon SNS and SQS.

We'll cover the following

Fanout is a common messaging pattern used in distributed architectures, such as event-driven architectures, where messages from one producer are sent to multiple consumers for asynchronous processing. In the fanout pattern, we have the following components:

  • Producers: Producers are the entities that send messages or events that need to be processed.

  • Exchange point: The messages from a producer are sent to an exchange point, which is responsible for delivering the messages to their respective destinations.

  • Queues: Messages sent from an exchange point are stored in a queue.

  • Consumers: Consumers fetch messages from a queue and process them.

Fanout using SNS and SQS

Solution architects can use Amazon SNS and Amazon SQS to create a fanout pattern in their cloud applications. Amazon SNS uses a push mechanism to deliver time-critical messages to its subscribers, while Amazon SQS uses a polling method to allow users to fetch messages from a queue whenever they are available.

Amazon SNS can be used as an exchange point, and Amazon SQS can be used to store messages to build a fanout messaging pattern. Any messages Amazon SNS receives from a producer will immediately be sent to its subscribers. We can add multiple SQS queues as subscribers to our SNS topic, which will store these messages and deliver them whenever a consumer makes a polling request.

Press + to interact
Fan-out architecture
Fan-out architecture

In the diagram above, a producer sends a message to an SNS topic, which duplicates it and delivers it to multiple SQS queues. The SQS queue then sends these messages to Lambda functions for further processing.

Use case: Processing objects uploaded in a bucket

Let’s assume we want to receive an email notification whenever an object is uploaded to an S3 bucket. Also, if the object being uploaded is an image, we want to add a watermark to it and extract any text that might be available in the image. These results will then be uploaded in another bucket.

We can build this application using Amazon SNS and SQS.

Press + to interact
Processing objects uploaded in a bucket
Processing objects uploaded in a bucket

The following steps are performed in the diagram given above:

  1. A user uploads an image in an S3 bucket.

  2. The S3 bucket generates an event and sends a message to an SNS topic. This message includes the bucket’s name, the object’s key, and metadata.

  3. SNS fanouts this message to its email and SQS queue subscribers.

  4. The consumer polls their SQS queue to fetch the message sent by their queue to process the file uploaded.

Benefits of using SNS and SQS

There are multiple benefits of using Amazon SNS and SQS together. Some of these are given below:

  • Decoupling components: Using SNS and SQS together, we can decouple the producers and consumers in our application. The producers can send messages to the SNS topic without worrying about data loss, as all messages will be stored in the SQS queue and will be fetched by the consumer whenever they’re available.

  • Message durability: The messages stored in SQS queues are retained until they are processed by their consumers. This ensures messages are not lost even if there is some failure at the consumers’ end.

  • Parallel processing: Using the fanout messaging pattern allows multiple consumers to process a message in parallel.

Get hands-on with 1300+ tech skills courses.