A nightly job pulls records from a SaaS platform's API, which returns JSON over HTTPS in pages of 100 records with a next-page token and responds with HTTP 429 when a client exceeds its request quota. The job misses records and often fails with 429 errors. Which fix is best?
Select an answer to reveal the explanation.
Short Explanation
An API that pages its results is like a book handing you one chapter at a time: you've got to keep asking for the next one until it says "the end." And when it says 429, "slow down," you wait and try again. Hammering it with parallel calls just earns you more 429s.
Full Explanation
Paginated APIs return a limited number of records per call and include a cursor or token that identifies the next page. A client that reads only the first page silently misses everything beyond it, which explains the missing records. The correct pattern loops, passing each next-page token until the response indicates no further pages. HTTP status 429 means Too Many Requests: the client has exceeded the provider's rate limit. Robust clients respond by pausing, ideally honoring any Retry-After value the server supplies or using exponential backoff, then retrying, so the job completes within the quota rather than failing. Requesting only the first page with a wider date range still caps the result at one page, so even more records would be missed. Opening many parallel connections increases the request rate, which triggers more 429 responses and may get the client blocked. Scraping the web interface instead is fragile, typically violates the platform's terms of use, and is usually subject to throttling and bot detection as well; when a documented API exists, it is the supported path. Exam caveat: treat API limits as part of the integration's design and capacity planning, not as an error to suppress. Operational check: log the number of pages and records fetched per run, compare the total to the source's reported count, and alert when retries are exhausted.