---
title: Direct AI crawlers with managed robots.txt · Cloudflare bot solutions docs
description: Protect your website or application from AI crawlers by implementing a robots.txt file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.
lastUpdated: 2025-04-29T15:53:32.000Z
source_url:
  html: https://a7ef1bf3.preview.developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/index.md
  md: https://a7ef1bf3.preview.developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
---

Protect your website or application from AI crawlers by implementing a `robots.txt` file on your domain to direct AI bot operators on what content they can and cannot scrape for AI model training.

Cloudflare's managed `robots.txt` explicitly disallows known bots engaged in scraping for AI purposes and AI agent activity.

AI bots are expected to follow the `robots.txt` directives. Otherwise, they risk getting banned.

To implement a `robots.txt` file on your domain based on your plan:

* Bot Fight Mode

  1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
  2. Go to **Security** > **Bots**.
  3. Select **Configure Bot Fight Mode**.
  4. On **Manage bot traffic with robots.txt**, select the toggle.

* Super Bot Fight Mode

  1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
  2. Go to **Security** > **Bots**.
  3. Select **Configure Super Bot Fight Mode**.
  4. On **Manage bot traffic with robots.txt**, select the toggle.

* Bot Management for Enterprise

  1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account and domain.
  2. Go to **Security** > **Bots**.
  3. Select **Configure Bot Management**.
  4. On **Manage bot traffic with robots.txt**, select the toggle.

Note

When you enable the `robots.txt` file on your domain, existing `robots.txt` files will be replaced.

## Availability

Managed `robots.txt` for AI crawlers is available on all plans.
