Why is my Webflow site blocked for bots and why is the sitemap verification in Google Search Console not working?

Your Webflow site may be blocked for bots if it is unintentionally set to prevent search engines from indexing it. This can also cause sitemap verification issues in Google Search Console.

1. Check Webflow’s SEO Settings

• Go to Project Settings > SEO tab in your Webflow dashboard.
• Scroll to the "Indexing" section.
• Make sure “Disable Webflow subdomain indexing” is unchecked (only affects subdomain).
• More importantly, ensure “Disable search engine indexing” is unchecked. If this is checked, your site sends a robots.txt rule blocking all crawlers.

2. Inspect robots.txt Configuration

• Webflow auto-generates a robots.txt file.
• If the above setting disables indexing, your site will serve:
  ```
  User-agent: *
  Disallow: /
  ```
  This blocks all bots, including Googlebot.
• To verify, visit yourdomain.com/robots.txt and confirm it doesn’t say "Disallow: /" unless intentional.

3. Verify Sitemap Is Accessible

• Your sitemap should be located at yourdomain.com/sitemap.xml.
• Check this URL in your browser. It must load correctly without a 404 or 403 error.
• If your site is password protected (via Webflow’s Site Password feature), the sitemap will be blocked for bots.
• Also, make sure that “Submit sitemap” in Google Search Console uses the correct format: just sitemap.xml, not a full URL.

4. Ensure DNS & Site Is Live

• Make sure you’ve published your Webflow site to your custom domain, not just the Webflow subdomain.
• If DNS isn’t pointed correctly or the site isn't fully live, Google can’t index or verify the sitemap.

5. Remove Password Protection

• In Project Settings > General, check if "Require site-wide password" is enabled.
• If so, bots (including Google) can’t access any page or the sitemap.
• Uncheck this if you intend the site to be public and indexable.

6. Confirm Site Isn’t in Development or Unpublished

• Only published domains are accessible by bots.
• If you've only published to yoursite.webflow.io, and not your custom domain, make sure the sitemap you submit corresponds to the correct domain.

Summary

To fix bot blocking and sitemap issues in Google Search Console:

• Enable search engine indexing in Webflow’s SEO settings.
• Remove any disallow rules in robots.txt.
• Confirm sitemap.xml loads correctly and is submitted properly.
• Remove any password protection from the live site.
• Make sure the site is published to your custom domain and is publicly accessible.

Once fixed, allow Google time to re-crawl the site and reprocess the sitemap.