The Problem
Async Lambda invocations — triggered by S3 events, SNS notifications, EventBridge rules, and similar sources — are fire-and-forget from the caller’s perspective. AWS will retry a failed invocation twice by default, but once those retries are exhausted, the event payload is silently discarded. There is no error surfaced to the original producer, no record of what was lost, and no built-in mechanism to replay it.
In practice this means a transient downstream failure — a database connection timeout, a third-party API blip, a cold-start memory spike — can permanently lose business-critical data. You often won’t notice until a downstream count is off or a customer reports a missing record. By then, the original event is gone.
A dead-letter queue changes that equation entirely. Failed payloads land in SQS where they sit for up to 14 days, visible and replayable. The rest of this post covers how to wire that up and how to replay messages safely once the underlying issue is resolved.
Approach
The pattern is straightforward: attach an SQS queue as the DLQ destination on the Lambda function. When Lambda exhausts its async retry attempts, it writes the original event payload to that queue automatically. Your team then has a durable, inspectable record of every failure.
One important boundary condition to understand before you start: the Lambda DLQ configuration only applies to asynchronous invocations. If your Lambda is itself triggered by an SQS event source mapping, the DLQ on the function has no effect on those messages — the source SQS queue’s own redrive policy handles failures in that case. Attaching a function-level DLQ to an SQS-triggered Lambda and expecting it to catch processing failures is a common misconfiguration that creates confusing behavior.
For the replay side, we will write a Python script that reads messages from the DLQ in batches, re-invokes Lambda synchronously with the original payload, and deletes the message from the queue only on confirmed success. Failures stay in the DLQ for a second look rather than being silently dropped again.
Wiring the Dead-Letter Queue to Lambda
Start by creating the SQS queue that will serve as the DLQ. Set MessageRetentionPeriod to 1209600 seconds (14 days) — this gives your team enough time to investigate and replay without racing against message expiry.
Once the queue exists, attach it to the Lambda function. The relevant configuration key is DeadLetterConfig.TargetArn. Via the AWS CLI:
aws lambda update-function-configuration \
--function-name myFn \
--dead-letter-config TargetArn=arn:aws:sqs:us-east-1:123456789012:myFn-dlqIf you are managing this in Terraform, the equivalent block inside your aws_lambda_function resource is:
dead_letter_config {
target_arn = aws_sqs_queue.dlq.arn
}The Lambda execution role also needs sqs:SendMessage permission on the DLQ queue. Without it, Lambda will silently fail to write to the DLQ and you will be back to square one. Add a policy attachment that grants this explicitly:
{
"Effect": "Allow",
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-1:123456789012:myFn-dlq"
}Verify the configuration took effect by describing the function and checking the DeadLetterConfig field in the response:
aws lambda get-function-configuration \
--function-name myFn \
--query 'DeadLetterConfig'Writing the Replay Script in Python
The script lives at scripts/replay_dlq.py and is intentionally stateless — all configuration comes from environment variables so the same script works across staging and production without modification. Set DLQ_URL and FUNCTION_NAME before running; BATCH_SIZE and MAX_BATCHES have sensible defaults.
The key design decision here is using InvocationType='RequestResponse' rather than async invocation during replay. This means the script waits for Lambda to return before deciding whether to delete the message. If the function errors again, the message stays in the DLQ and the failure is printed immediately — no silent re-queuing, no guessing.
# scripts/replay_dlq.py
# Reads messages from an SQS dead-letter queue and replays them to Lambda.
# Usage: DLQ_URL=<url> FUNCTION_NAME=<name> python replay_dlq.py
import os
import json
import boto3
DLQ_URL = os.environ["DLQ_URL"]
FUNCTION_NAME = os.environ["FUNCTION_NAME"]
BATCH_SIZE = int(os.environ.get("BATCH_SIZE", "5"))
MAX_BATCHES = int(os.environ.get("MAX_BATCHES", "20"))
sqs = boto3.client("sqs")
lam = boto3.client("lambda")
def receive_batch():
response = sqs.receive_message(
QueueUrl=DLQ_URL,
MaxNumberOfMessages=BATCH_SIZE,
WaitTimeSeconds=5,
AttributeNames=["All"],
)
return response.get("Messages", [])
def replay_message(message):
try:
payload = json.loads(message["Body"])
except json.JSONDecodeError:
print(f"[SKIP] Non-JSON body: {message['MessageId']}")
return False
response = lam.invoke(
FunctionName=FUNCTION_NAME,
InvocationType="RequestResponse",
Payload=json.dumps(payload),
)
status = response["StatusCode"]
func_error = response.get("FunctionError")
if status == 200 and not func_error:
print(f"[OK] Replayed {message['MessageId']}")
return True
else:
print(f"[FAIL] {message['MessageId']} — FunctionError: {func_error}")
return False
def delete_message(message):
sqs.delete_message(
QueueUrl=DLQ_URL,
ReceiptHandle=message["ReceiptHandle"],
)
def main():
replayed, failed, skipped = 0, 0, 0
for _ in range(MAX_BATCHES):
messages = receive_batch()
if not messages:
print("No more messages in DLQ.")
break
for msg in messages:
success = replay_message(msg)
if success:
delete_message(msg)
replayed += 1
else:
failed += 1
print(f"\nDone — replayed: {replayed}, failed: {failed}, skipped: {skipped}")
if __name__ == "__main__":
main()The IAM identity running this script needs three permissions: sqs:ReceiveMessage and sqs:DeleteMessage on the DLQ, and lambda:InvokeFunction on the target function. Scope these tightly — there is no reason this script needs broader Lambda or SQS access than those three actions.
Handling Poison Pills and Retry Limits
A DLQ without a redrive policy on the DLQ itself is incomplete. Consider what happens when a message in the DLQ has a malformed payload that will never succeed — the replay script will skip it (the json.JSONDecodeError path returns False without deleting), but a human still needs to deal with it eventually.
More critically, if you are using the DLQ queue as a source for any other consumer, you need a redrive policy with maxReceiveCount set between 3 and 5. This prevents a bad message from cycling indefinitely through a consumer that keeps failing on it. Configure this on the SQS queue directly:
aws sqs set-queue-attributes \
--queue-url https://sqs.us-east-1.amazonaws.com/123456789012/myFn-dlq \
--attributes '{
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:myFn-dlq-poison\",\"maxReceiveCount\":\"3\"}",
"MessageRetentionPeriod": "1209600"
}'This creates a two-tier structure: the primary DLQ catches Lambda async failures, and a secondary “poison pill” queue catches messages that even the DLQ consumer cannot process after three attempts. Messages in the poison queue warrant direct inspection — they typically indicate a structural payload problem rather than a transient error.
In the replay script, the [SKIP] path for non-JSON bodies is intentionally non-destructive. Those messages remain in the DLQ so you can inspect the raw body via the SQS console or CLI before deciding whether to discard or manually correct them.
Verify It Works
The fastest way to confirm the wiring is correct is to deploy a Lambda function that intentionally raises an exception, invoke it asynchronously, and then check the DLQ for the resulting message.
Invoke the function asynchronously and wait for Lambda to exhaust its two built-in retries (this takes a few minutes with default settings):
aws lambda invoke \
--function-name myFn \
--invocation-type Event \
--payload '{"test": "trigger_failure"}' \
/dev/nullAfter the retries complete, check whether the message arrived in the DLQ:
aws sqs get-queue-attributes \
--queue-url https://sqs.us-east-1.amazonaws.com/123456789012/myFn-dlq \
--attribute-names ApproximateNumberOfMessagesYou should see ApproximateNumberOfMessages increment to 1. If it stays at 0, the most likely causes are a missing sqs:SendMessage permission on the Lambda execution role, or the function being invoked synchronously rather than asynchronously.
Once the message is confirmed in the DLQ, fix the Lambda function code, redeploy it, and run the replay script:
DLQ_URL=https://sqs.us-east-1.amazonaws.com/123456789012/myFn-dlq \
FUNCTION_NAME=myFn \
python scripts/replay_dlq.pyA successful run prints [OK] for each replayed message and ends with a summary line. Check ApproximateNumberOfMessages again — it should be back at 0, and the function’s CloudWatch logs should show the event was processed correctly.
Related Ideas
Lambda Destinations are worth considering alongside DLQs. Destinations support both success and failure routing, and they pass richer metadata including the request and response payloads. If you need to route failures to different targets based on error type, or if you want to record successful invocations as well, Destinations give you more control than a DLQ alone. The two are not mutually exclusive — you can configure both on the same function.
EventBridge Pipes provide a cleaner path for replay if your architecture already uses EventBridge. A Pipe can read from the SQS DLQ and route messages back through an EventBridge bus, which then re-triggers the original Lambda. This removes the need for a manual replay script and makes the recovery path part of your infrastructure definition rather than an operational runbook.
SNS fanout is relevant when multiple consumers need to react to the same failure event. Rather than routing DLQ messages to a single Lambda for replay, you can publish to an SNS topic that fans out to a replay Lambda, an alerting Lambda, and an archiving Lambda simultaneously. This is useful in audit-heavy environments where you need a permanent record of every failure independent of the replay outcome.
Finally, if you find yourself writing replay scripts frequently, it may be worth centralising the pattern into an internal tool that accepts queue URL and function name as arguments and handles pagination, rate limiting, and structured output consistently. The script in this post is a solid starting point for that kind of abstraction.
