AWS DataSync Task Failures: Finding the Error the Console Hides

Your DataSync task shows FAILED in the console, there’s no error message anywhere in the UI, and your data pipeline is silently stale. The CloudWatch dashboard looks clean. The SNS alert fired, but the body just says TaskExecutionFailed with no detail. You’re staring at a black box.

This is the runbook I wish I’d had the first time this happened to us at 2 AM on a Sunday.

Symptoms — What a Broken DataSync Task Looks Like

Before touching anything, confirm you’re actually dealing with a failure and not just a slow run. These are the signals worth checking in order:

• CloudWatch metric FilesTransferred drops to 0 mid-run — not at start, mid-run. That’s the tell. A task that never starts failing is a different problem.
• Task status stuck in LAUNCHING or RUNNING for more than two hours — DataSync doesn’t have a built-in timeout that surfaces cleanly. It just sits there.
• SNS alert fires with TaskExecutionFailed — no error code, no detail. Just the event name. This is AWS’s default behavior and it’s genuinely unhelpful.
• Console UI shows no error message — this is by design. The console surfaces task-level status, not execution-level error detail. Most engineers stop here and start guessing.

To distinguish a slow task from a hung one, pull BytesTransferred from CloudWatch and look at the rate over a 15-minute window. If it’s consistently zero for 30+ minutes during an expected active window, the task is hung, not slow.

Root Cause — Why DataSync Fails Without Telling You

In my experience, silent DataSync failures fall into three categories. Check them in this order — it saves time.

1. IAM role misconfiguration. The most common cause. Either the execution role is missing datasync:*, or the destination S3 bucket policy doesn’t allow PutObject for the role. Watch out for this one: if you see "Access Denied: Unable to assume role" buried in execution logs, the problem is the trust policy — datasync.amazonaws.com is missing as a principal. That’s not an S3 permissions issue, even though it reads like one. I’ve seen engineers spend an hour tightening bucket policies when the trust relationship was the actual problem.

2. VPC endpoint or agent connectivity loss. The DataSync agent requires outbound TCP 443 to datasync.<region>.amazonaws.com. Port 80 is only needed during initial activation — after that it can be closed. A security group change that blocks 443 will silently kill in-progress transfers.

3. Source share permissions and agent version mismatch. NFS/SMB shares can reject the agent user after a permissions change. Separately — and this one burned us — agents older than version 2.0 (released March 2021) silently drop files larger than 5 GB on NFS sources. No error. Files just don’t transfer. The upgrade path is AMI replacement, not an in-place update.

There’s also a throttling scenario worth knowing: 503 SlowDown errors from S3 get buried in CloudWatch Logs and are never surfaced as task errors. You’ll only find them if logging is actually enabled — which it isn’t by default.

Fix #1 — Restore Visibility with CloudWatch Logs

Before changing any config, get the actual error message. You can’t fix what you can’t see.

Step 1: Enable task-level logging. The default LogLevel for new DataSync tasks is OFF. Nothing is written to CloudWatch unless you explicitly set it to BASIC or TRANSFER. Set it to TRANSFER for full per-file visibility during troubleshooting. The log group is /aws/datasync.

⚠️ Gotcha: If you’re managing DataSync tasks in Terraform and the log_configuration block references a log group that doesn’t exist yet, task creation fails silently. Pre-create the log group or add an explicit depends_on.

Step 2: Pull the execution error detail directly from the CLI. This is the command most engineers never run because the console looks complete. It isn’t.

This command returns the ErrorCode and ErrorDetail fields that the console never shows you:

aws datasync describe-task-execution \
  --task-execution-arn arn:aws:datasync:us-east-1:123456789012:task/task-abc123/execution/exec-xyz789 \
  --region us-east-1 \
  --output json \
  | jq '{Status, ErrorCode: .Result.ErrorCode, ErrorDetail: .Result.ErrorDetail}'

Step 3: Query CloudWatch Logs Insights for the actual transfer errors. Use this query against /aws/datasync:

fields @timestamp, @message
| filter @message like /ERROR|WARN/
| sort @timestamp desc
| limit 50

Once you have a real error message, you can stop guessing and start fixing.

Fix #2 — Repair IAM and Network Connectivity

The two most common root causes, fixed without redeploying the task.

This is the minimum IAM policy for a DataSync execution role. The logs:* permissions are required if you enabled logging in Fix #1 — missing them causes a secondary silent failure:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3TransferPermissions",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::your-destination-bucket",
        "arn:aws:s3:::your-destination-bucket/*"
      ]
    },
    {
      "Sid": "CloudWatchLogsPermissions",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/datasync:*"
    }
  ]
}

The trust policy must include datasync.amazonaws.com as a principal — this is the part that gets missed:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "datasync.amazonaws.com"  // Must be present — missing this causes "Unable to assume role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

For network connectivity, check agent status first:

aws datasync describe-agent \
  --agent-arn arn:aws:datasync:us-east-1:123456789012:agent/agent-abc123 \
  --region us-east-1 \
  | jq '{Status, LastConnectionTime}'

Status: OFFLINE means the agent can’t reach the DataSync service endpoint. Check your VPC security group for outbound TCP 443 to datasync.us-east-1.amazonaws.com. There’s no default CloudWatch alarm for agent offline status — you have to build that yourself, and most teams don’t.

I stopped using a single shared DataSync IAM role across multiple tasks after one bucket policy tightening broke six unrelated tasks simultaneously. One role per task boundary is worth the overhead.

Fix #3 — Handle Partial Transfers and Recover Data Integrity

After a failed execution, the concern is what got transferred and what didn’t.

First, enable task reports to get per-file status. Add TaskReportConfig to your task pointing to a dedicated S3 bucket. The report lands at s3://<bucket>/datasync-reports/<execution-id>/ and lists every file with its transfer result. The bucket needs a policy granting datasync.amazonaws.com the s3:PutObject permission — missing this silently disables reports with no warning.

Second, fix your transfer mode. TransferMode: ALL re-copies every file on every run regardless of modification time. This is the default in many example configs and it’s expensive. On a 20 TB dataset, a re-run with TransferMode: ALL costs roughly $250 per execution at $0.0125/GB. Switch to TransferMode: CHANGED with OverwriteMode: NEVER to only move what’s actually new.

Third, the runbook script. This script inspects a task execution, emits a custom CloudWatch health metric, and pulls the last 20 error lines from logs in one pass. Run it from your incident response workflow or wire it to an EventBridge rule:

#!/usr/bin/env bash
# datasync-check.sh — Runbook script: inspect a DataSync task execution
# and emit a custom CloudWatch metric for health tracking.
# Usage: ./datasync-check.sh <task-execution-arn> <region>
# Requirements: AWS CLI v2, jq
# IAM needed: datasync:DescribeTaskExecution, cloudwatch:PutMetricData, logs:FilterLogEvents

set -euo pipefail

EXECUTION_ARN="${1:?Usage: $0 <task-execution-arn> <region>}"
REGION="${2:?Provide AWS region}"
NAMESPACE="Custom/DataSync"
METRIC_NAME="TaskExecutionHealth"

echo "==> Describing task execution..."
RESULT=$(aws datasync describe-task-execution \
  --task-execution-arn "$EXECUTION_ARN" \
  --region "$REGION" \
  --output json)

STATUS=$(echo "$RESULT" | jq -r '.Status')
ERROR_CODE=$(echo "$RESULT" | jq -r '.Result.ErrorCode // "NONE"')
ERROR_DETAIL=$(echo "$RESULT" | jq -r '.Result.ErrorDetail // "No error detail"')
FILES_TRANSFERRED=$(echo "$RESULT" | jq -r '.Result.TransferredCount // 0')
FILES_FAILED=$(echo "$RESULT" | jq -r '.Result.ErrorCount // 0')
BYTES_TRANSFERRED=$(echo "$RESULT" | jq -r '.BytesTransferred // 0')

echo "==> Status:            $STATUS"
echo "==> Error Code:        $ERROR_CODE"
echo "==> Error Detail:      $ERROR_DETAIL"
echo "==> Files Transferred: $FILES_TRANSFERRED"
echo "==> Files Failed:      $FILES_FAILED"
echo "==> Bytes Transferred: $BYTES_TRANSFERRED"

# Map task status to a numeric health value for CloudWatch custom metric
# 1 = healthy (SUCCESS), 0 = any other state (FAILED, LAUNCHING stuck, etc.)
if [[ "$STATUS" == "SUCCESS" ]]; then
  HEALTH_VALUE=1
else
  HEALTH_VALUE=0
  echo "==> ALERT: Task not in SUCCESS state. Investigate the error detail above."
fi

echo "==> Pushing health metric to CloudWatch namespace: $NAMESPACE"
aws cloudwatch put-metric-data \
  --region "$REGION" \
  --namespace "$NAMESPACE" \
  --metric-name "$METRIC_NAME" \
  --value "$HEALTH_VALUE" \
  --unit "None" \
  --dimensions "ExecutionArn=$EXECUTION_ARN"

LOG_GROUP="/aws/datasync"
EXECUTION_ID=$(basename "$EXECUTION_ARN")  # Extracts execution ID from full ARN

echo "==> Querying CloudWatch Logs for errors (last 20 lines)..."
aws logs filter-log-events \
  --region "$REGION" \
  --log-group-name "$LOG_GROUP" \
  --filter-pattern "ERROR" \
  --log-stream-name-prefix "$EXECUTION_ID" \
  --max-items 20 \
  --query 'events[*].message' \
  --output text 2>/dev/null \
  || echo "WARNING: No log stream found. Verify LogLevel != OFF on the task."

echo "==> Done."

Prevention — Alarms and Baselines That Actually Page Someone

The operational gap I see most often: teams monitor task status but never establish a baseline for expected duration. A task that normally runs in 45 minutes and is now at hour six doesn’t trigger anything. The data goes stale. Nobody notices until a downstream report is wrong.

Add these four things to your DataSync runbook:

1. Alarm on TaskExecutionFailed count > 0 with a 1-minute evaluation period. Route by task env and owner tags. Without tags, all failures go to one queue and nobody owns them.

2. Alarm on BytesTransferred = 0 for 30 consecutive minutes during the expected run window. This catches hung tasks that never formally fail.

3. Alarm on agent offline status. Wire an EventBridge rule to call describe-agent on a schedule and push a custom metric. There is no native alarm for this — you have to build it. See the kuryzhev.cloud DevOps runbooks for an EventBridge-to-Lambda pattern that handles this cleanly.

4. Set a duration baseline per task. Track TaskExecutionDuration as a custom metric over 30 days. Alarm when a run exceeds 150% of the rolling average. A 6-hour overrun on a task that normally takes 40 minutes is a failure signal even if the task hasn’t formally errored yet.

For the full DataSync monitoring reference, the AWS DataSync monitoring documentation covers available CloudWatch metrics. For task report configuration details, see the task reports reference.

The pattern here is simple: DataSync is quiet when things go wrong. Your job is to make it loud before the data pipeline owner calls you.