Skip to content
WebrenewDocs

Publish and get found

Control how crawlers discover your site

Set crawler preferences, publish a sitemap, and understand what these controls do and do not protect.

Who this helps
Owners, Admins, and Editors preparing a website for public discovery.
Before starting
Connect a custom domain and decide which pages should appear in search.
Done when
The published robots.txt, sitemap.xml, and optional llms.txt reflect the intended settings.

01

Choose crawler access

Open Site overview → Crawling. The robots.txt options express preferences to crawlers that follow them.

Allow all permits crawlers. Block AI crawlers allows search engines while asking known AI crawlers to stay out. Block all asks every crawler to stay out.

  1. Choose the intended Crawler access option.
  2. Use Custom only when there is a specific robots.txt requirement.
  3. Save, then publish the site.
  4. Open /robots.txt on the custom domain and inspect the result.

robots.txt is not access control. It does not make a public page private or require a visitor to sign in.

02

Check search visibility separately

SEO & Social controls whether search engines should show the site or a page. Crawling controls access preferences.

Turning off a site's search hiding does not override Block all in Crawling. Both settings must match the intended public launch.

03

Publish the page list

The Sitemap switch controls the site's sitemap.xml. Launch rebuilds the sitemap on publication.

A hidden or blocked site can show that its sitemap is off even when the switch is enabled.

  1. Enable Publish a sitemap.
  2. Check site visibility and crawler access.
  3. Publish the site.
  4. Open /sitemap.xml on the custom domain.
  5. Submit that address in Search Console if appropriate.

A sitemap helps discovery. It does not guarantee that a search engine will index a page or rank it.

04

Choose an llms.txt source

llms.txt is an optional text description of the site's pages. Choose Made from your pages, Off, or Custom.

The automatic version uses page titles and descriptions and updates on publication. It is separate from AI crawler blocking.

  1. Choose the source in the llms.txt card.
  2. If Custom, check every address and statement in the text.
  3. Save and publish.
  4. Open /llms.txt on the custom domain when enabled.

An llms.txt file does not guarantee inclusion in AI answers, traffic, or permission enforcement.

← All guides