Google Using Generic & Undocumented Google Crawler (www.seroundtable.com)
from tonytins@pawb.social to technology@lemmy.world on 11 Aug 17:43
https://pawb.social/post/29619682

#technology

threaded - newest

LodeMike@lemmy.today on 11 Aug 19:29 next collapse

The crawler user agent is just named “Google,”

skeezix@lemmy.world on 11 Aug 20:05 collapse

Very sneaky.

Toes@ani.social on 11 Aug 21:48 next collapse

Probably just a misconfigured bot and nothing special.

Flyswat@lemmy.dbzer0.com on 11 Aug 22:00 collapse

Can we get rid of the current model where websites are financed by ads and have instead these AI companies pay the websites they crawl?

I could really do with less ads if they are going to crawl the web anyway.