Webflow sync, pageviews & more.
NEW

Why is my Webflow site blocked for bots and why is the sitemap verification in Google Search Console not working?

TL;DR
  • Check and update robots.txt in Webflow SEO settings to allow crawling.
  • Disable "Disable indexing" and remove password protection if enabled.
  • Ensure site is published to a custom domain and sitemap URL is accessible at yourdomain.com/sitemap.xml.
  • Fix any 403 or 404 errors by verifying sitemap URL and site accessibility.
  • Use Google Search Console's "Inspect URL" tool to request re-indexing after changes.

Your Webflow site may be blocked for bots if certain settings or files prevent search engine access, leading to Google Search Console sitemap verification issues.

1. Check Robots.txt Settings

  • In Webflow, go to Project Settings > SEO.
  • Look for the robots.txt input box.
  • If you see lines like User-agent: * Disallow: /, this blocks all bots from crawling your site.
  • Solution: Remove or modify these rules to allow crawling. A basic crawl-friendly version is:
  • User-agent: *
  • Disallow:
  • Save the changes and re-publish the site.

2. Ensure Site Is Not Password-Protected or Disabled for Indexing

  • In Project Settings > General, check the “Disable indexing” option. If this is enabled, Webflow adds a "noindex" meta tag and blocks bots.
  • Also make sure the site is not password protected, which would prevent bots from accessing any content.
  • Solution: Uncheck both settings if you want your content to appear in search engines.

3. Confirm Sitemap URL and Hosting Status

  • Webflow automatically generates a sitemap at yourdomain.com/sitemap.xml.
  • Make sure you've published your site to a custom domain and not just the Webflow staging domain (e.g., yoursite.webflow.io).
  • In Google Search Console, submit the sitemap as sitemap.xml under your verified custom domain.
  • Visit https://yourdomain.com/sitemap.xml in your browser to ensure the sitemap loads.

4. Check for 403 or 404 Errors on Sitemap Submission

  • If Google Search Console shows a 403 Forbidden or 404 Not Found error:
  • Double-check the sitemap URL.
  • Make sure the site is published and accessible.
  • If it's a 403, it could be bots blocked by either robots.txt or server settings.

5. Wait for Google to Reprocess Changes

  • After updating robots.txt, sitemap, or indexing settings, Google may take some time to reprocess.
  • You can use the “Inspect URL” tool in Google Search Console to manually request indexing.

Summary

Your Webflow site may be blocking bots due to a restrictive robots.txt, “Disable indexing” setting, or an incorrectly submitted sitemap. Fix these by allowing crawlers in SEO settings, ensuring the site is live and not hidden from indexing, and verifying the correct sitemap URL submission.

Webflow Resources
Webflow templates
Webflow inspiration
Rate this answer

Other Webflow Questions

Explore Logic tutorials
How can I fix the issue with my grid layout sticking out on the right side of the page on tablet breakpoint in Webflow?
Has anyone implemented code to automatically redirect visitors to country-specific websites in Webflow? For example, if a person located in Denmark visits the .no site, can they be redirected to the dk site?
Is there a way to convert Sketch artboards to Webflow or set up a Sketch layout that aligns with the Webflow grid system?
How can I link from a button on page 1 to a specific tab on page 2 in Webflow, which is located inside a section and needs to scroll down the page?
How can I activate a modal with a "subscribe" button in Webflow or have the modal automatically pop up after a set amount of time?
How can I adjust and move items on a Webflow website if I am unable to resize or reposition them?
Can I disable email submission for a form on Webflow? I have multiple forms on my site and want to prevent one specific form from sending emails, as it is a zip code submission that redirects to an external booking page.
Does Webflow have mobile functionality for adding to your CMS streams via mobile, or is there a feature in the works? If not, are there any other solutions or experiences with making Webflow CMS edits from mobile?
Where can I start to resolve an 'INP issue' with mobile views for a Webflow site flagged by Google search console?
How can I easily pull custom parameters from a page URL and replace them with placeholders in Webflow?
Is there a way to replicate a Google map style drag to pan and scroll to zoom on an image in Webflow? Is it possible to achieve this without using interactions? I'm trying to create a webmap for a fantasy game project and want users to be able to pan and zoom high resolution rasterized images of our world. I've considered using Mapbox, but it seems too complex. Are there any other options available in Webflow?
How can I dynamically update the meta title and description on a Webflow site using JavaScript? I want to display the updated information from a collection list on the page, but I am unsure if this will affect SEO.
How can I fix the issue where the other divs do not move downward when content is expanded in Webflow?
How can I add anchor links to different subtopics of a CMS page in Webflow?
Is it possible to duplicate a scroll interaction on a navbar/logo from one page to another in Webflow?
Can Webflow be used to call a CSS or JS file stored on a different server, instead of hosting them on GitHub?
How can I dynamically redirect to a CMS created after form submission in Webflow?
In Webflow, how can I hide specific elements in the footer (such as "Ramin Khaze The Difference" and "Schedule showing") when the URL contains a certain parameter, such as "?mls"? I am a beginner in coding and would appreciate any help. Thank you!
How can I prevent the "w-current" class in Webflow from overriding the height of my custom class on a link block element?
Can anyone help me add a digital clock and date to my Webflow site? I've been able to add custom code, but I need assistance with styling the text. Thank you in advance for your help!
Can I capture a screenshot of a Webflow page on click and convert it to a downloadable PDF?
How do I mirror user input from a form field to a non-form text field on the same Webflow page in real-time using JavaScript?
How can I close a dropdown in Webflow when clicking on a link?
Is there a more optimal way to create a portfolio section with a sticky image container that changes as you scroll, similar to the design on Deveb's website, that works across different web browsers, especially Safari, using Webflow or any other JavaScript tools?
How can I implement a multi-select field with a pre-selected option in a popup form on a collection page using Webflow?
How can I alternate the layout of the image and heading for each collection item in a two-column format on Webflow?
How can I execute JavaScript code when clicking a specific button with a given ID in a Webflow project?
How can I disable the Webflow success or failure state for a sign-up form and display a custom thank you page using jQuery and the Webflow form submit state?
How can I create a vertical scrolling menu with Webflow, similar to the one on Apple's website, that switches to horizontal scrolling when the menu doesn't fit on one screen?
How can I change the language of the Google Maps embed from English to Arabic in Webflow?