IndexNow in 60 lines of bash — and the two traps that eat an afternoon

seoindexnowbashwebdev

IndexNow is the rare piece of SEO infrastructure that is actually just an HTTP request. You host a key file at your site root, POST a list of URLs, and one request reaches Bing, Yandex, Naver, Seznam and Yep — the API fans it out.

Google does not participate. Nothing in this post affects Google. Say that out loud before you start, because it is the single most common misunderstanding and it determines whether IndexNow is worth your time at all.

I submitted 22,818 URLs this way. Twelve requests, every one HTTP 200. Here is the whole thing, plus the two mechanical traps that cost me the first two attempts.

The protocol

  1. Generate a key: 8–128 hex characters.
  2. Serve it as plain text at https://yoursite.com/<key>.txt, containing exactly the key and nothing else.
  3. POST JSON to https://api.indexnow.org/indexnow:
{
  "host": "yoursite.com",
  "key": "00cbf1ff17dc47824f373896bd042679",
  "urlList": ["https://yoursite.com/a", "https://yoursite.com/b"]
}

Up to 10,000 URLs per request. That’s it. No account, no dashboard, no OAuth.

Trap 1: curl: Argument list too long

A batch of 2,000 URLs is roughly 150 KB of JSON. The obvious thing —

curl -X POST "$ENDPOINT" -H 'Content-Type: application/json' --data "$body"

— dies before a single request leaves the machine. $body becomes an argv entry, and 150 KB blows past the exec argument limit. The error names curl, so you spend twenty minutes suspecting curl.

Write the body to a file and hand curl the file:

curl -X POST "$ENDPOINT" \
  -H 'Content-Type: application/json; charset=utf-8' \
  --data-binary "@body.json"

Trap 2: a split glob that silently double-submits

split -d names its output chunk.00, chunk.01, … So you loop:

split -l 2000 -d urls.txt chunk.
for c in chunk.[0-9]*; do            # <-- wrong
  jq ... < "$c" > "$c.json"
  curl ... --data-binary "@$c.json"
done

The loop writes chunk.00.json — which also matches chunk.[0-9]*. Depending on glob expansion timing you submit some batches twice, and there is no error, because double-submitting is perfectly valid. You just quietly burn your rate limit and wonder why the counts don’t line up.

Glob exactly the digits split produces:

for c in chunk.[0-9][0-9]; do

Trap 3, the expensive one: an unserved key file

If the key file isn’t reachable, every batch returns 403. A 403 from IndexNow means “key invalid” — and a key that is perfectly valid but was never deployed produces the identical response. I regenerated the key twice before checking whether the file was actually being served.

Two things to get right:

  • The file must contain the key and nothing else. A trailing newline makes it 33 bytes instead of 32. Write it with printf '%s', not echo.
  • Verify it over HTTP before submitting anything, and refuse to proceed if it doesn’t match.
served="$(curl -fsS "$ORIGIN/$KEY.txt" || true)"
[ "$served" = "$KEY" ] || {
  echo "key file not served at $ORIGIN/$KEY.txt (got: '${served:0:40}')" >&2
  exit 1
}

That check turns a confusing afternoon into one clear line of output.

The script

#!/usr/bin/env bash
set -euo pipefail

HOST="yoursite.com"
ORIGIN="https://$HOST"
BATCH=2000   # protocol allows 10000; smaller bodies, clear of the rate limiter

# Derive the key from whatever <hex>.txt is in public/ — rotating the key is
# then "drop in a new file, delete the old one" with no code change.
KEY="$(basename "$(ls public/[0-9a-f]*.txt | head -1)" .txt)"

served="$(curl -fsS "$ORIGIN/$KEY.txt" || true)"
[ "$served" = "$KEY" ] || { echo "key file not served" >&2; exit 1; }

urls="$(mktemp)"; trap 'rm -f "$urls" "$urls".*' EXIT
curl -fsS "$ORIGIN/sitemap.xml" \
  | grep -o '<loc>[^<]*</loc>' | sed 's|</\?loc>||g' > "$urls"

split -l "$BATCH" -d "$urls" "$urls".
for chunk in "$urls".[0-9][0-9]; do
  jq -Rn --arg host "$HOST" --arg key "$KEY" \
    '{host: $host, key: $key, urlList: [inputs]}' < "$chunk" > "$chunk.json"
  code="$(curl -s -o /dev/null -w '%{http_code}' \
    -X POST "https://api.indexnow.org/indexnow" \
    -H 'Content-Type: application/json; charset=utf-8' \
    --data-binary "@$chunk.json")"
  printf '%5d urls  HTTP %s\n' "$(wc -l < "$chunk")" "$code"
  sleep 3
done

jq -Rn '... [inputs]' is the neat bit: -R reads raw lines, -n suppresses the default input, and [inputs] slurps every line into an array — so the URL list is built without any shell quoting of 2,000 strings.

Two design choices worth stealing

Don’t wire this into your deploy script. Mine reads the already-published sitemap over HTTP rather than the local build output. That means it runs independently of a deploy and cannot break one — and your deploy script is load-bearing in a way a submission script is not. Run it after a deploy, from cron, or by hand.

Derive the key from the filesystem, don’t hardcode it. ls public/[0-9a-f]*.txt means rotating the key is dropping in a new file and deleting the old one. A hardcoded key is a code change plus a deploy plus a chance to get them out of order.

If you’re on Cloudflare, check before you build this

Cloudflare has a “Crawler Hints” toggle that does IndexNow for you. It is free and it is one click, so try it first — but measure it, because it derives its signal from cache status: it reports on MISS. If your site deliberately doesn’t edge-cache HTML (mine doesn’t — caching it would mean adding a purge step to the deploy, or a deploy leaves the old release live worldwide), there may be nothing for it to report. Check the IndexNow submitted-URL count in Bing Webmaster Tools after a deploy. If it’s near zero, Crawler Hints isn’t working for you, and the script above is the answer.

What to actually expect

IndexNow is a discovery push, not an indexing guarantee. It tells participating engines a URL exists. Whether they crawl it, and whether they index what they crawl, is entirely their call.

It is also worth being honest about who’s on the other end. Bing matters — it also feeds DuckDuckGo, Yahoo and Ecosia. Yandex matters if you have Russian-language traffic. Naver and Seznam both openly deprioritise foreign-language content, so if your site is English and you aren’t targeting Korea or Czechia, expect roughly nothing from them despite the free submission.

And Google, the one you actually care about, isn’t in the list at all.

From the deploy pipeline behind navyduck.com — 22,818 static pages of certification practice questions.

← All articles