LLM.txt & AI Crawler Setup Guide for Recruitment agencies

Create a text file at /recruiter.txt with a brief introduction of your recruitment agency's specializations (e.g., IT staffing, executive search).

Include markdown-style links to your most important service pages, candidate portals, and employer branding content.

Add a 'Candidate FAQs' and 'Client FAQs' section to address common queries from AI models acting as job seekers or hiring managers.

User-agent: * (or specific bot like GPTBot, ClaudeBot) Allow: /jobs/ Allow: /talent-glossary/ Allow: /industry-reports/ Disallow: /internal-admin/

Verify your crawler permissions using tools that simulate AI bot access to your site.

Monitor crawl frequency in your server logs to ensure AI bots are accessing your core talent and market data pages, not just boilerplate.

Wrap individual job descriptions in `<article>` tags to denote distinct opportunities.

Use `<section>` with descriptive `aria-label` attributes for key job components like 'Responsibilities', 'Qualifications', and 'Benefits'.

Ensure all skill matrices or salary data tables use proper `<thead>` and `<tbody>` tags for structured data extraction by AI.

Keep related job requirements or candidate skill sets within distinct content blocks (e.g., 500 words) for easier RAG processing.

Avoid ambiguous phrasing; explicitly state the primary job title or required skill in summaries of each section.

Eliminate vague pronouns ('it', 'they') and replace them with specific job titles, company names, or skill sets to ensure clarity for AI.