A records-purge job lists thousands of S3 inventory objects but stops halfway because the code ignores continuation tokens. What should the developer fix?
Select an answer to reveal the explanation.
Short Explanation
Big S3 listings come like a binder with page tabs—miss a tab and you only purged half the shelf. Walk every continuation token with the SDK's paginator (and keep retries on). Glacier storage class, a longer timeout alone, or HeadBucket won't finish a multi-page list.
Full Explanation
A records-purge job that lists thousands of S3 inventory objects but stops halfway because the code ignores continuation tokens must implement SDK pagination (and standard retries) so every continuation page is processed. ListObjects and ListObjectsV2 return truncated pages by design; without following NextContinuationToken (or the older Marker), municipal purge jobs silently leave objects behind—exactly the compliance failure records managers fear when retention schedules say “delete after N years.” Pagination is not optional polish; it is correctness for any list API at city scale.
Switching the bucket to S3 Glacier Instant Retrieval does not make listing return everything in one call—storage class does not remove page limits. Raising only the Lambda timeout without fixing how list results are consumed still processes a single page and then exits “successfully” while orphans remain. Replacing ListObjects with a single HeadBucket call only checks that the bucket exists; it discovers no keys and cannot drive a purge.
Exam caveat: pagination patterns repeat across AWS list APIs (S3, DynamoDB Scan/Query, IAM, EC2 describe calls). Pair page loops with retries/backoff for transient 503s during large listings. For enormous buckets, prefer S3 Inventory reports over live full-bucket listing when the use case allows.
Operational check: loop while IsTruncated is true and pass NextContinuationToken on each call; unit-test with mocked multi-page responses; retry throttling errors; log pages processed and object counts; and reconcile purge counts against inventory so silent early exits cannot pass unnoticed.