Processing SNS Messages

Learn how messages are published, delivered, and monitored in Amazon SNS.

We'll cover the following

The data sent from a producer to an SNS topic in Amazon SNS is known as a message. This message is then distributed to all of the subscribers of the topic. Let’s look at the different features Amazon SNS provides us to publish and receive messages.

Publishing messages

After creating an SNS topic and adding subscribers, we can publish messages to a topic using the AWS Management Console or the AWS SDK. The maximum size of the message we can publish is 256 KB, however, we can extend this size by using the Extended Client Libraries for Java and Python

Using these libraries, we can extend the maximum message size to 2 GB. These libraries work by storing the actual payload of our message in an S3 bucket and sending the reference to this bucket to the subscribers of the SNS topic. Once the subscriber receives this reference, they can de-reference it using the client libraries and retrieve the original message sent by the publisher.

Amazon SNS also allows us to publish up to 10 messages in batches, rather than publishing one message at a time to an SNS topic. Through this, we can significantly reduce the cost of sending notifications in our applications.

Delivering raw messages

Amazon SNS provides the option of sending raw messages to ensure SQS, Kinesis Data Firehose, and HTTP/S endpoints don’t accidentally process the JSON formatting of the messages sent from a topic. If this option is enabled, any metadata SNS adds to a message during publishing or delivering a message is removed.

Following is an example of a message an SNS topic delivers to its subscribers when raw message delivery is disabled:

{
  "Type": "Notification",
  "MessageId": "dc1e94d9-56c5-5e96-808d-cc7f68faa162",
  "TopicArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:DemoTopic",
  "Subject": "Demo Message",
  "Message": "Hello! From Educative.",
  "Timestamp": "2024-01-01T21:41:19.978Z",
  "SignatureVersion": "1",
  "Signature": "FMG5tlZhJNHLHcccccccxxxxxxssssdddTuNYShn2h0bKNC/zLTnMyIxEzmi2X1rrrrrrrrttttttuuuuuWHEE73yDVR4SyYAikP9jrd5O7lX1c1uvnxFZva9hNklVyPfy+7TalMD0lzmJuOrExtnSIbZew3foxgx8GT+lbZkLd0ZdtdRJlIyPRP44eyq78sU0Eo/LsDr0Iak4ZDpg8dXg==",
  "SigningCertURL": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-0000000111111333333bdb98bd93083a.pem",
  "UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:xxxxxxxxxxxx:DemoTopic:e1039402-0000-1111-abcd-797da162b297"
}
Message delivered to subscribers when raw message delivery is disabled

The message given below will be received by the topic’s subscribers if raw message delivery is enabled:

Hello! From Educative.
Message delivered to subscribers when raw message delivery is enabled

Filtering messages

By default, Amazon SNS forwards all messages published to a topic, to the topic’s subscribers. However, we can provide filter policies to ensure only a subset of the messages being published are sent to a subscriber.

In a filter policy, we create a JSON object that defines the type of messages a subscriber receives. This can be done on the basis of message attributes or message body. When SNS receives a message from a publisher, it forwards it to the subscriber. However, if a filter policy is associated with the subscriber, SNS first compares the message with the policy and forwards this message only if the message doesn't violate the policy.

Press + to interact
Message filtering in Amazon SNS
Message filtering in Amazon SNS

In the diagram given above, the Lambda function does not receive the messages m2 or m3.

Monitoring messages

The messages published to an SNS topic can be monitored to ensure sensitive information like personally identifiable information (PII) can be audited, masked, or blocked. This is done by using data protection policies.

Data identifiersData identifiers are a collection of techniques and criteria, such as machine learning patterns, used to detect sensitive information in our messages are used in data protection policies to scan our messages for sensitive information. Amazon SNS provides us with pre-managed data identifiers to detect PII and allows us to create custom identifiers according to our use case. Once a data identifier is found in a topic, we can configure SNS to take one of the following actions:

  • Audit: In this operation, the message is sent to the topic’s subscribers, however, SNS logs its findings in S3 buckets, Kinesis Data Firehose, or CloudWatch logs.

  • De-identify: In this operation, sensitive information is masked or removed from the message before forwarding it to the topic’s subscribers.

  • Deny: In this operation, SNS blocks the Publish request or doesn't forward the message it receives to the topic’s subscribers.

Get hands-on with 1300+ tech skills courses.