The backups were failing every night and nothing said so

The frightening thing about backups is not that they fail. It’s that they fail politely. A backup plan sits in the console looking healthy, the schedule is green in the sense that it exists, and the jobs it launches have been dying quietly every night for a week. Nobody notices, because the failure mode of a backup is silence, and silence is exactly what a working backup looks like too.

I’ve stopped being surprised by this. The expensive incidents are almost never the dramatic ones. They’re the boring gap between “we configured a something” and “that thing is actually doing its job,” and nobody owned the difference.

The symptom

A daily plan was set up to snapshot a handful of EFS filesystems. On paper, fine. In reality every scheduled job had been failing with the same line:

Your backup job failed as AWS Backup does not have permission to describe
resource. Review your IAM policies.

A permissions error, on a plan that referenced the standard role name. So the role “should” have been there. It was not.

The root cause was nothing clever

aws iam get-role --role-name AWSBackupDefaultServiceRole
# An error occurred (NoSuchEntity) ...

The role did not exist. That’s the whole bug. And the single most useful habit here is reading the error precisely: NoSuchEntity is not AccessDenied. One means the thing is genuinely absent; the other means your own principal is blocked. People burn afternoons debugging a permissions problem they don’t have because they skimmed the error. Backup was assuming a role that had never been created, so every job failed at the first step. No role, no permissions, no backups, no noise.

The constraint that made it interesting

In a well-run enterprise account you often can’t just click “create the default role.” You’re inside a locked-down IAM setup, and what you can do is a short list. Mine was:

Action	Allowed?
iam:CreateRole	yes
iam:AttachRolePolicy	yes
iam:GetRole	yes
iam:DeleteRole	no
iam:DetachRolePolicy	no

Create but not delete. That is a genuinely awkward place to stand, and it’s more common than the tutorials admit. It means every role you make is permanent until someone with more rights removes it, so you stop poking around and start being deliberate. Which, honestly, is how you should have been operating anyway.

The fix, and the one detail that will waste your afternoon

# 1. create the role. The PATH is not decoration. See below.
aws iam create-role \
  --role-name AWSBackupDefaultServiceRole \
  --path /service-role/ \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": { "Service": "backup.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }]
  }'

# 2. attach the two AWS-managed policies (backup + restore)
aws iam attach-role-policy --role-name AWSBackupDefaultServiceRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForBackup
aws iam attach-role-policy --role-name AWSBackupDefaultServiceRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForRestores

# 3. don't wait for tonight's schedule; seed one now
ROLE=arn:aws:iam::111122223333:role/service-role/AWSBackupDefaultServiceRole
aws backup start-backup-job \
  --backup-vault-name my-backup-vault \
  --resource-arn arn:aws:elasticfilesystem:ap-southeast-2:111122223333:file-system/fs-0abc123 \
  --iam-role-arn "$ROLE"

The detail: the plan’s selections referenced role/service-role/AWSBackupDefaultServiceRole. Create the role at the default path (/) and its ARN is role/AWSBackupDefaultServiceRole, which does not match. Backup keeps failing with the identical error and you’ll swear you fixed it. The path is part of the identity, not a filing convenience. Create it at /service-role/ and everything the plan already points at simply lines up.

Once the role existed, the on-demand job completed and the nightly schedule started passing on its own. No redesign. Someone just had to look.

The part almost everyone skips

A backup you have never restored is a hope, not a backup. I ran a full round trip, and it found two problems in about twenty minutes that would otherwise have surfaced during an actual outage, which is the worst possible time to learn them.

The shape of an honest restore test:

1. Write a sentinel into the source: a unique string, a small known table,

and a checksum of a real file.

1. Take an on-demand backup and wait for the recovery point.
2. Restore to a brand new throwaway filesystem. Never over production.
3. Mount it and verify the sentinel and checksum match byte for byte.
4. Delete the throwaway and remove the sentinel.

Gotcha one: encrypted restore demands the key. First attempt failed at once:

Required key(s) [kmskeyid] missing

If the source is encrypted, the restore metadata must carry the KMS key id.

aws kms describe-key --key-id alias/aws/elasticfilesystem \
  --query KeyMetadata.Arn

Gotcha two: your data isn’t where you left it. AWS Backup restores EFS content under a generated subdirectory like /aws-backup-restore_<timestamp>/, not at the mount root. Don’t assume the original path. Search for the file:

find /mnt/restored -name 'prod.db'

After that the checksum matched the baseline exactly. Recoverability proven, throwaway deleted. Total cost: a few cents and one afternoon, versus finding out during a real recovery.

The boring conclusion

Silent backup failure is the default, not the exception. A plan is a claim; a completed job is evidence; a tested restore is proof. Read your errors precisely, treat the role path as identity, and accept that operating inside tight IAM just means being deliberate. Then put a recurring reminder on the restore test, because the first one finds the bugs and the repeat one is what lets you stop thinking about it.

# is tonight's run actually green?
aws backup list-backup-jobs --by-backup-vault-name my-backup-vault \
  --query "BackupJobs[?State!='COMPLETED']"
# empty is the answer you want