How can I whitelist the IP or user agent of a third-party vendor to crawl my Webflow site for legal compliance archiving purposes?
To whitelist the IP or user agent of a third-party vendor to crawl your Webflow site for legal compliance archiving purposes, you can follow these steps:
1. Identify the IP or User Agent: Get the specific IP address or user agent of the third-party vendor that needs access to crawl your site. You can contact the vendor directly to obtain this information.
2. Access your Webflow Project Settings: Log in to your Webflow account and go to the Project Settings of the site you want to whitelist the IP or user agent for.
3. Go to the Hosting tab: In the Project Settings, navigate to the Hosting tab, where you will find options related to your site's hosting settings.
4. Scroll down to the "SEO" section: In the SEO section, you'll find the option to add custom code to your site's `
` tag.5. Open the Custom Code section: Click on the "Custom Code" button within the SEO section to open the editor.
6. Add the whitelist code: To whitelist the IP, you can use a code snippet like this in the Custom Code editor:
```
```
Replace `'Vendor User Agent'` with the actual user agent string provided by the vendor, and `'${request.ip}'` with the specified IP address.
7. Save and Publish: After adding the whitelist code, click the "Save Changes" button to apply the changes to your site. Then, republish your site using the "Publish" button to make the changes live.
By implementing this code, you are allowing access to the specified IP or user agent, and restricting others from crawling your site for compliance archiving purposes. Make sure to double-check the IP or user agent details provided by the vendor to ensure accuracy.
It's worth noting that allowing access to specific IPs or user agents should be done cautiously, as granting access to improper entities could potentially compromise the security or integrity of your website. Always verify the legitimacy of the vendor and consult with legal professionals if needed.