New install on new subdomain worse than the old subdomain
from peter@deddit.petersanchez.com to azorius on 24 Aug 2023 17:24
https://deddit.petersanchez.com/g/azorius@azorius.net/p/V78w1ZX24Gjn4VkZKm-New-install-on-new-subdomain-worse-than-the-old-

So I finally got tired of the hit and miss of using azorius on the same subdomain as my old lemmy install. I assumed it was a key issue so I killed the DB and re initialized the build, and am now hosting it under the new subdomain deddit.

I subscribed to all the same communities but this time I noticed when loading a new community no posts were shown, though the logs clearly show json dumps with posts from those groups. I didn't think anything of it and kept subscribing to my groups. Sometimes a couple of posts would show but usually zero posts.

Today, I see a few commeints but no new posts. Even though the groups are active with new posts, etc.

I see a lot of these entries in the logs:

2023/08/24 16:02:45 no post for comment: https://programming.dev/comment/2291160

Anything I can do to troubleshoot this? Are there any key files stored I should have wiped out or something? I did delete the entire -datadir and recreated it.

#azorius #sad-n-confused

threaded - newest

peter@deddit.petersanchez.com on 24 Aug 2023 17:26 next collapse

Eyy, at least I see this posted correctly to the azorius.net group :)

tedu on 24 Aug 2023 19:16 next collapse

That's a pretty normal log entry. We don't (currently) attempt to fetch parents for comments. When you subscribe to a new group, you'll get a bunch of comments for posts you don't have, but as the group rolls over and new posts come in, everything should work.

I think you just need to wait a bit longer. If you're getting comments, even if we're not currently saving them, that's a good sign everything is working. The posts will get there eventually.

The slow start is not ideal, but I don't think it indicates anything wrong.

tedu on 24 Aug 2023 22:03 next collapse

How did you follow these groups? When you search for the group, it should pull the outbox. It skips this if you search for one post. But prog@azorius is empty. I didn't think that was possible. And I wrote it. :)

You found a bug! 🎉 Recent change to treat background refresh nicer ended up breaking things.

peter@deddit.petersanchez.com on 25 Aug 2023 17:03 next collapse

Hah I knew it! Because I got like 4 new posts overnight when there had to have been 100+ for all my groups.

So yea, I search for the group by full url,it loads the page (but shows no posts, which didn't happen before) and I click the subscribe button. Logs show successful post to remote group inbox (which didn't work before on old subdomain).

So is the fix pushed? Will I need to unsubscribe and resubscribe?

tedu on 25 Aug 2023 18:34 collapse

Everything should be good. The existing groups will backfill some posts after a day or two as well. Just make sure you have change 1e0138409da4.

peter@deddit.petersanchez.com on 25 Aug 2023 19:22 next collapse

OK, updated. I'll watch it over the coming days. Thanks!

peter@deddit.petersanchez.com on 28 Aug 2023 15:53 next collapse

So is there some like polling that happens after X time? Because I notice I do get new stuff but it's bursts by groups. So like, the first 10 most recent are from a single group, the next 10 are from another group, etc. It seems like things aren't being pushed to me but pulled from Azorius.

tedu on 29 Aug 2023 18:21 collapse

Yes. Every day or so, we poll group outboxes looking for posts that we missed somehow. This also helps fill in groups that don't have any subscribers, and would appear dead otherwise.

The code is in refresh.go.

peter@deddit.petersanchez.com on 29 Aug 2023 18:56 next collapse

Thanks! I'll keep an eye on things but it seems I have a delay of up to 24 hours for any new posts in groups I'm subscribed to. I keep referring to world@lemmy.world because it's a pretty active group but it seems anything pushed to me isn't sticking. I'll dig around the logs a bit too and update if I find anything odd.

peter@deddit.petersanchez.com on 29 Aug 2023 19:20 collapse

After a big of digging through logs it seems it's lemmy.world that is not posting updates to me. Trying to unsubscribe produces this error:

2023/08/29 19:16:01 failed to post json to https://lemmy.world/c/selfhosted/inbox: http post status: 403

Are the keys stored in the database (azorius.db) or on disk? I did destroy the db and re-create it when moving to a new sub domain but maybe the keys survived if they're stored on disk somewhere other than the db? Could that be the issue (though I guess it doesn't make sense it works with other lemmy instances)

tedu on 29 Aug 2023 19:29 collapse

That's a little weird, but lemmy.world is always having problems. :(

In my testing, a bad key would result in a 400 status.

The keys are stored in the database. Every instance will have new ones.

You may be stuck for a bit in the delivery retry queue. After a failed response like 403, we'll save that and try it again for a while. But new messages will line up behind the failed message. So effectively, you can't talk to lemmy.world until the 403 message times out. This is by design, but if there's a federation problem, it's not great.

You can peek in the database at the deliveries table, and delete anything with a rcpt of lemmy.world. Then try subscribing again.

peter@deddit.petersanchez.com on 07 Sep 2023 20:49 collapse

Well, I just went and unsubscribed and resubscribed and finally I got 200 OK responses and Accept posts back. So maybe things are all worked out now. Wish I could say what the issue or resolution was but as the saying goes, time heals all wounds.

tedu on 29 Aug 2023 19:49 collapse

Your trouble aside, what I like about this post is it inspired me to write the sub to post feature so I'd get notified of comments, and then you replied and I didn't get the notification because I mixed up the self logic.

peter@deddit.petersanchez.com on 30 Aug 2023 00:00 collapse

Hah, happy to be of service