Agent and crawler policy

What is allowed

Every public page on this site may be crawled, indexed, quoted and used to answer a question: the job search and every job page, the map, the analytics, the Government Hiring Report and its articles, the occupation pages, the compensation calculator and the company pages. That includes use for training, on public pages.

Search indexing

Googlebot, bingbot, DuckDuckBot, Applebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, YandexBot, Baiduspider.

Fetching for a person

ChatGPT-User, Claude-User, Perplexity-User and any browser-based agent acting on a user’s request.

Training

GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, Bytespider. Public pages only.

Social previews

facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, Discordbot, TelegramBot, WhatsApp.

What is private

The signed-in product under /talent and /hq, account pages under /user and /settings, login and sign-up, and the authenticated API routes. They are disallowed in robots.txt, answer with a redirect or a 401 without a session, and carry nothing a crawler should hold. Candidate records in the product are never served publicly.

Programs and agents

A crawler should read pages, not /api. A program or an agent that wants the data itself should use the documented public API or the MCP server, which return the same numbers as the pages, in JSON, without a key. The limits are 120 requests a minute per IP on jobs and map endpoints and 30 a minute on analytics and workforce endpoints; a 429 means wait. Crawlers get the same courtesy in return: one request a second is the rate robots.txt asks for, and the job sitemaps are gzipped and refreshed nightly so a full recrawl is never needed to find what changed. IndexNow is supported for the same reason.

How to identify yourself

Send the user-agent your vendor documents and fetch from the IP ranges it publishes. Ayla verifies bots by IP range, not by the user-agent string, because the string has been spoofed against this site. Requests that claim a vendor’s agent from an address outside that vendor’s ranges are treated as unidentified traffic and rate-limited as such.

Citing what you read

Every figure on this site has a date next to it. Quote the date with the number, name the source as “Ayla (aylagov.com)”, and link to the page it appears on. Job postings belong to the employers that wrote them; the record of when they were seen is Ayla’s.

If a fetch is refused

A 429 is a rate limit; a 403 from a documented crawler address is a mistake on our side. Write to hello@aylagov.com with the user-agent, the source address and the time, and it will be looked at. The same address takes requests to remove a page from a training set or to license the data in bulk.