New Threat Research Reveals AI crawlers make up almost 80% of AI bot traffic, Meta Leads AI Crawling As ChatGPT Dominates Real-Time Web Traffic (www.fastly.com)
from Pro@programming.dev to technology@lemmy.world on 22 Aug 02:04
https://programming.dev/post/36106846

cross-posted from: programming.dev/post/36083939

#technology

threaded - newest

PattyMcB@lemmy.world on 22 Aug 02:48 next collapse

I couldn’t click the cookie policy garbage because the AI chat bot obscured the buttons on that page.

Talk about AI slop

dan@upvote.au on 22 Aug 03:54 next collapse

Are they sure that the Meta traffic is AI bots? A lot of people share links in posts on Meta products, which scrape the links to generate a preview.

morto@piefed.social on 22 Aug 04:15 next collapse

Aren't those requests made from client devices?

dan@upvote.au on 22 Aug 04:29 collapse

They are for anything that’s E2E encrypted such as Messenger and WhatsApp, so that the link isn’t revealed to Meta servers.

As far as I know, for things like posts on Facebook, the server does the scraping. This is especially the case in the web version, since client-side scraping of arbitrary sites would require those sites to have an open CORS policy, which comes with security risks.

There’s a Sharing Debugger tool on the Facebook developer site that lets you force rescrape a site.

bert_macklin_fbi@lemmy.world on 22 Aug 04:23 collapse

It would not surprise me to see that the traffic traced back to “scrapers”. A user’s perceived relevance often ties to the recency of the dataset.

friend_of_satan@lemmy.world on 22 Aug 06:32 next collapse

Soooo what’s the other 20% if not crawlers?

INHALE_VEGETABLES@aussie.zone on 22 Aug 09:37 next collapse

It’s me replying

Grostleton@lemmy.dbzer0.com on 22 Aug 17:52 collapse

3 Indian tech workers in a trench coat.

3dcadmin@lemmy.relayeasy.com on 23 Aug 08:24 collapse

Meta actually leading the way with something, however negative it is!