LLM.txt & AI Crawler Setup Guide for Forums

Create a text file at /community.txt with a brief introduction to your forum's purpose and primary topics.

Include markdown-style links to your most important categories, sticky threads, and official announcements.

Add a 'FAQ' section within the file to address common queries about forum rules, user roles, or technical support for AI agents.

Implement `User-agent: LLM-Bot Allow: /category/general-discussion/ Allow: /category/technical-support/ Disallow: /private-messages/ Disallow: /user/settings/

Verify your crawler permissions and access patterns using tools like `crawl-test.com` or by monitoring server access logs for the specified user-agent.

Monitor crawl frequency and depth in your server logs to ensure LLM bots are accessing relevant discussion threads and not overwhelming user-specific or administrative sections.

Wrap individual discussion threads or 'topics' within `<article>` tags to signal their primary content.

Use `<section>` with descriptive `aria-label` attributes for distinct forum categories or sub-forums (e.g., `aria-label='Software Development Discussions'`).

Ensure all user-generated content, especially structured data within posts (e.g., code snippets, bug reports), uses appropriate semantic tags like `<code>`, `<pre>`, and adheres to Schema.org markup for `DiscussionForumPosting`.

Keep related posts, replies, and their context within a logical container of approximately 500-750 words to facilitate effective chunking.

Avoid 'floating' context by ensuring each post or reply explicitly references the main topic or preceding post it's replying to, using clear identifiers.

Eliminate ambiguous pronouns (e.g., 'It,' 'They,' 'This') and replace them with explicit references to the product, feature, or user being discussed to improve RAG accuracy.