Generating Unique References

Before you start working on the second function, there’s an important technical aspect of working with S3 that you need to consider, which will also come into play frequently when using other storage systems. S3 isn’t really an atomic database; it’s eventually consistent. That means it’s not safe to check whether a file name exists and, if not, then upload something there. You need some way of creating unique references to avoid naming conflicts. A typical solution for this would be to use some kind of unique ID generator, but then you have a problem with traceability. It’s not easy to correlate logs and traces with outputs.

Each Lambda request has a unique request ID, which is automatically printed into the logs. You can read it from the second argument of the Lambda function, using context.awsRequestId. This is a great candidate for unique file names or message identifiers created from a Lambda function. Using a reference based on the request ID makes it unique but also easy to correlate with a processing session. Even more importantly, if the Lambda runtime retries processing an event after an error, it will use the same request ID. So if the function dies half-way through working and Lambda spins up another container to recover from the error, it won’t generate two different files.

Using AWS resources from Lambda functions #

You can use the AWS SDK in the Lambda function to talk to other AWS resources, including S3. Normally, when using the AWS SDK, we need to provide authentication details when instantiating the API. When using the AWS SDK inside a Lambda function, you can just instantiate the service objects without providing credentials, like the following: