A blogger found in early November that pages published after 7 October had stopped appearing in Yandex search. Articles were going out three times a week on schedule, were present in the sitemap, IndexNow was returning 202, and YandexBot was appearing in nginx logs daily. A url: query on any new article returned nothing, and the situation persisted for a month.
The fix came from the recrawl submission button in Yandex Webmaster. On 10 November, 10 URLs were submitted manually; the crawler fetched 9 of them the next day and 7 showed 'appeared in search' in the API event log. Daily manual submission was impractical, so a script was written instead.
When the API returns 504
The POST /v4/user/{user_id}/hosts/{host_id}/recrawl/queue/ endpoint returned 504 Gateway Timeout on every address submitted. The rest of the Webmaster API — indexed page lists, events, queries — responded normally throughout. Because of this, the script drives a browser rather than calling the endpoint directly.
How the automation works
playwright-cli connects to a Chrome profile where the author had already signed into Yandex once by hand. The script has no access to the password. launchd runs it at 10:45 Moscow time, 15 minutes after the day's article is published by the server.
URLs are drawn from three sources: the article published that day, a manual queue (a plain text file with slugs of rewritten posts), and articles absent from search according to the API. Addresses sent within the last 14 days are skipped when pulling from the third source, but are always taken from the manual queue. Each URL is checked with a HEAD request before submission to avoid spending one of the daily slots on a 301 redirect.
An earlier version set the textarea value via JavaScript and fired a synthetic input event. The field filled but the submit button stayed greyed out. Playwright's fill command resolved this — the button activates the same way it does on a real keystroke. Button state is checked via b.disabled in the page itself rather than from the accessibility snapshot, which was found to misrepresent a disabled button.
A --dry-run flag runs every step except the final click and prints what would have been submitted. It proved useful on the first live day, 12 November, when the daily limit was already spent. That same morning the Mac's DNS was not resolving, which took down the API source, but the manual queue was sent in full. The DNS failure was never explained.
Results and constraints
The first live run on 12 November accepted 10 of 10 URLs. Over three days, including two manual submissions, 30 addresses were sent. Pages indexed in Yandex rose from 41 to 45; the numbers are not clean because some articles were rewritten in the same period and Yandex had been dropping them as low-value.
Moving the script to the server that runs the blog's other automation was ruled out: signing into Yandex from a headless browser in a data centre would likely trigger security checks, and an earlier attempt to transfer a saved session between machines on a different service ended badly. If the Mac is asleep at 10:45, launchd will run the missed job when it wakes.