Cloudflare Will Filter Out Web Crawlers That Serve AI Companies


The hosting platform wants sites to have more control over how AI companies use their content.

Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s stance is becoming more defensive by default.

“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Matthew Prince, Cloudflare’s CEO and co-founder shared in a statement. “Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training.”

Web traffic used to indicate that people were viewing a website’s ads or paying for its subscriptions, but the popularity of AI models that can visit sites on a user’s behalf to pull up-to-date information has upended that system. Cloudflare’s new approach is an attempt to rebalance the relationship in a way that’s fair for both AI companies and anyone running a website.

Starting September 15, 2026, new customers and new websites from existing Cloudflare subscribers will default “to allow for search but block training and agent use for pages with ads.” Mixed-use crawlers that don’t give site owners the option to choose whether their site is used for AI will also be blocked on pages with ads by default. Users with free accounts will also switch to these defaults unless they opt-out ahead of the September 15 deadline, according to the company.

As part of these changes, Cloudflare is also releasing a new version of the Pay Per Crawl feature it introduced in 2025 that allowed websites to block AI web crawlers by default unless companies paid to scrape their content. The feature is now called Pay Per Use, and rather than base payments on whether a webpage has been crawled, Cloudflare says site owners will be paid when their content appears in answers from AI chatbots. The announcement only mentions partnerships with Ceramic.AI and You.com, but Cloudflare likely hopes other AI companies will join as its customers opt in.

Besides generally trying to make the relationship between websites and AI companies more fair, as TechCrunch notes, Cloudflare also seems to be indirectly targeting Google. The company’s announcement mentions that “the largest search engine has access to about 2X more information than leading AI companies because they make it difficult for customers to remain discoverable without also being used for AI.” Google’s main crawler, Googlebot, both indexes websites for the company’s various search engines and collects information to train Gemini and power AI features like AI Overviews and AI Mode. Google lets websites opt-in to a separate crawler called Google-Extended that only crawls websites for traditional search results, but if a publisher wanted to be included in AI Mode results, but doesn’t want their content to train Google’s models, they don’t have an option. Cloudflare’s new policy is an attempt to force Google and other companies with mixed-use crawlers to change their tactics.



Source link

Leave a Reply

Subscribe to Our Newsletter

Get our latest articles delivered straight to your inbox. No spam, we promise.

Recent Reviews



Medically reviewed by Robert Burakoff, MD

Drinking herbal tea and winding down with a relaxing activity can help soothe your gut at night.Credit: Drazen Zigic / Getty Images
Drinking herbal tea and winding down with a relaxing activity can help soothe your gut at night.
Credit: Drazen Zigic / Getty Images
  • Small habits in the evening can influence your gut health.
  • Eating a Mediterranean-style dinner, drinking a cup of herbal tea, and taking an after-dinner walk are simple ways to support your gut health in the evening.
  • Stress management, a regular bedtime routine, and getting seven to nine hours of sleep each night help anchor your circadian rhythm, which may keep your gut bacteria balanced.

Gut health is something you can work at improving at all hours of the day. Here are seven simple habits you can do in the evening hours to support gut health.

1. Have a Mediterranean Dinner

Following the basics of the Mediterranean diet at dinner could support your gut health in the evening and beyond. This includes:

  • Fresh fruits and vegetables
  • Legumes
  • Whole grains
  • Poultry
  • Fish
  • Healthy fats, like olive oil

The foods you eat determine the makeup of your gut microbiome, or the system of microorganisms that live in your gastrointestinal tract. Whole foods and minimally processed foods contain nutrients, such as fiber, vitamins, and minerals, that support a balanced gut.

The Mediterranean diet is known for its health benefits, including reduced inflammation and improved overall health. Research also shows that the diet has a positive impact on the gut. People who followed a Mediterranean diet were found to have more microorganisms in their gut compared to people who ate a diet higher in sugar, fat, and salt.

2. Finish Eating 2-3 Hours Before Bed

Eating too close to bedtime could disrupt your sleep and impact gut health. A meal that isn't fully digested can cause unpleasant digestive symptoms, especially if you're prone to indigestion. While there are no rules about when to stop eating before bed, finishing your last full meal two to three hours before bedtime can give your body enough time to digest.

New, early research suggests that eating late at night may negatively affect the gut. The researchers found connections between stress, late-night eating, and bowel issues. Those who had both high stress and late-night eating patterns were 2.5 times more likely to also report bowel problems and have lower gut microbiome diversity.

If you need a snack after dinner, opt for foods that have protein, fiber, or healthy fats, including:

  • Fruits, like berries or kiwi
  • Low-fat yogurt
  • Nuts
  • While grains
  • Fermented foods like kimchi, kefir, or kombucha

3. Take a Walk After Dinner

An evening stroll can have benefits beyond getting in your daily steps. It could also help your body move food through the digestive system.

You don't have to walk for that long, either. In one four-week study, researchers found that a 10-15-minute walk after a meal was more effective than a prokinetic medication for easing bloating and discomfort in adults who regularly experience bloating.

If you can't walk around the neighborhood, even standing up and moving your body around the room could help.

4. Have a Cup of Herbal Tea

Sipping a caffeine-free herbal tea in the evening can be a relaxing ritual. Certain teas, including ginger and peppermint tea, can also help with digestion and ease unpleasant symptoms such as gas and bloating.

Ginger is known for easing stomach discomfort. Drinking ginger tea can help with gas and bloating because ginger reduces constipation and the breakdown of undigested nutrients in the gut. You can buy dried ginger tea in a bag or make it from fresh ginger. It's generally considered safe to drink in moderate amounts daily.

Peppermint can help with cramping and bloating. The tea is made from peppermint leaves and is generally considered safe. One thing to note is that peppermint can also trigger indigestion, so if you're prone to heartburn, it might not be a good choice.

5. Pick a Wind-Down Practice

The brain and gut are connected. Stress affects gut health, and higher stress levels are linked to stomach issues such as nausea, constipation, and diarrhea. When your stress hormones are high, it can lead to increased gut inflammation and potentially an imbalance in gut bacteria.

Choosing a relaxing activity to practice in the evenings could help lower your stress, which can also benefit your gut. Try a mind-body or mental health practice, such as:

  • Meditation
  • Breathing exercises
  • Yoga
  • Journaling
  • Gratitude affirmations

Some evidence suggests that slow, deep breathing has positive impacts on the gut, such as improving symptoms in people with irritable bowel syndrome (IBS).

6. Keep Bedtime Consistent

Having a regular bedtime routine can help you get more consistent sleep, which may also positively affect your gut. Research suggests that changes to your sleep cycle can affect your gut microbiome. Keeping a regular sleep and wake time anchors the circadian rhythm. Even a few days of irregular sleep could impact your gut.

While researchers are still learning about the complex connection between the gut and sleep, it appears as though sleep disturbances can change the gut's makeup and function. Setting a regular bedtime and wake time and sticking to it as closely as possible could benefit your gut health.

7. Get Enough Sleep

Something as simple as regularly getting enough sleep could have a big impact on your gut health. Experts recommend that adults get between seven and nine hours of sleep each night. Sleeping less than seven hours could put you at risk for health issues.

Shorter sleep time can affect the gut microbiome, potentially causing an imbalance that could lead to inflammation and metabolic problems.

The relationship between sleep and your gut seems to work both ways. Just as poor sleep can negatively affect gut health, an unbalanced gut may contribute to sleep troubles. If you're not consistently getting the recommended amount of sleep and aren't able to improve your sleep with at-home habits, it might be worth asking a healthcare provider for help.



Source link