Post: Link to user bug?

Link: https://azorius.net/c/7bb2ZwJc3cYym3cQpF

tedu on 18 Aug 2023 19:39 collapse

I got tired of bots crawling the user pages for remote users. So anything that's @remote is 404 unless you're logged in. It should maybe not be linked, that's an oversight in the template. Have to find all the places.

Big picture, I'm still debating how to handle search engines and remote content. On the one hand, I think forum posts should be searchable, so don't want a blanket robots.txt. But I don't want to become the authority for remote content by accident. I think there's a <link rel> tag to be use. That should be good enough.

Other ideas I had were putting all remote content under an /x/ route, but it starts getting complicated using all the right urls in all the right places. Or maybe not? I think it's mostly abstracted. Could be done.

The other evil idea is to send back 404 status codes, but still return the content, so it looks like a normal page to browsers but engines will ignore it. I don't know if I want to open this rabbit hole.

So, short answer is, somewhere in the template, if you're not logged in, those user links should not be added.

But I'm curious where you see a link to tedu@azorius.net on azorius.net. That should not happen.