Has anyone experienced the issue of Googlebot being blocked by robots.txt in Webflow, even though access has been allowed in the file?

Googlebot being blocked despite being allowed in the Webflow-generated robots.txt can happen due to a few possible causes. Here's how to troubleshoot and resolve it.

1. Check the robots.txt File in Webflow Settings

• Go to Project Settings > SEO tab in your Webflow project.
• Scroll down to the robots.txt field.
• Make sure it includes a directive like:
• User-agent: *
• Allow: /
• Double-check that there are NO disallow rules for / or other critical paths.

2. Confirm What’s Actually Live

• Even if the Webflow settings look correct, publish your site again to ensure the latest robots.txt is live.
• Visit https://yourdomain.com/robots.txt directly in your browser to verify the contents.
• Sometimes old versions get cached. Force a browser refresh (Ctrl+F5) or check via terminal tools like curl.

3. Inspect Google Search Console Reports

• In Google Search Console, look for URLs that show “Blocked by robots.txt.”
• Use the URL Inspection Tool on a problematic page. It will tell you if crawling was denied and which rule caused it.

4. Consider Conflicting Webflow Features

• If your site or specific pages are set to “Disable indexing” in Webflow’s page settings:
• This adds a <meta name="robots" content="noindex"> tag but does not affect robots.txt.
• However, Google may still choose not to crawl due to this tag being present — though it will not be indicated as a robots.txt block.
• Check your Page Settings in the Designer to confirm indexing isn't disabled.

5. Make Sure You're Not Using Password Protection

• If your site is protected by Webflow’s Site-wide Password Protection:
• Webflow automatically blocks all crawlers via robots.txt.
• You must remove password protection (in Project Settings > Memberships & Passwords) to allow Googlebot access.

6. Check for Temporary Redirects or Deploy Errors

• If you just recently published or changed DNS settings, there might be a delay before Googlebot sees the updated file.
• Use the “Robots.txt Tester” in Search Console or run curl -I https://yourdomain.com/robots.txt to confirm the server response is 200 (OK).

Summary

If Googlebot is being blocked despite an “Allow” rule in Webflow’s robots.txt, check for live file mismatches, site-wide password protection, or incorrect Page-level settings. Publishing your site and verifying the live robots.txt is usually the first step to resolve this.