We've been having some stability issues over the past few days that I've tracked down to database performance, so I'll be doing some tuning and also applying all our needed system updates at the same time.

Cheers!

Update: This is all done now!

top 19 comments

sorted by: hot top controversial new old

[–] grte@lemmy.ca 20 points 4 months ago

Thanks for the notice and also for keeping this whole enterprise functional.

[–] djvinniev77@lemmy.ca 12 points 4 months ago

Thanks for your hard work.

[–] rbesfe@lemmy.ca 10 points 4 months ago

Thank you for all you guys do to keep this amazing instance running

[–] Vilian@lemmy.ca 9 points 4 months ago

[–] Mereo@lemmy.ca 8 points 4 months ago

Thank you for your awesome work!

[–] nik282000@lemmy.ca 7 points 4 months ago

Thanks for the heads up!

[–] Darkassassin07@lemmy.ca 6 points 4 months ago

We've been having some stability issues over the past few days

I was gonna ask about that. I had noticed several slow downs over the last couple days, but pretty brief. Glad to see it's being looked into/sorted out.

Chears, m8!

[–] veeesix@lemmy.ca 6 points 4 months ago* (last edited 4 months ago)

Thank you for the hard work! 💪

[–] avidamoeba@lemmy.ca 5 points 4 months ago

🙌

[–] cybermass@lemmy.ca 4 points 4 months ago (1 children)

PST? Do the lemmy.ca servers happen to be hosted in the Vancouver area?

I'm somewhat of a network specialist, though in wireless technology. Now I'm the head of my company's IT department so I'm a jack of all trades but would love to get some experience working on database infrastructure.

If you want some free labour, let me know how I can get in contact.

[–] Shadow@lemmy.ca 8 points 4 months ago (1 children)

Server is in Beauharnois, but I'm in Vancouver.

Unless you're already a postgres tuning expert, I think we're all good. Thanks for the offer though!

[–] cybermass@lemmy.ca 3 points 4 months ago

No worries, always looking for ways to help out the community.

Cheers!

[–] ininewcrow@lemmy.ca 4 points 4 months ago

Thanks! ... great work in maintaining everything ... it's much appreciated!

[–] remotelove@lemmy.ca 2 points 4 months ago* (last edited 4 months ago) (1 children)

Howdy! I know y'all probably have a plan for Saturday, but was hoping you could peek at another possible issue.

While I believe this might be a client issue (Connect), some communities from other instances are not being listed properly. So, being logged into lemmy.ca and then trying to list /c/cat on lemmy.world shows a strange list of posts that resembles /c/all.

This is not a widespread spread issue and may not happen when logged into other instances. (196 is another with a similar issue here, which is annoying as I prefer to monitor that community from this instance.)

Unfortunately, the Connect dev has been missing for a bit and we cannot verify where the issue actually is. (It would suck to switch client apps .. )

Thanks in advance!

[–] Shadow@lemmy.ca 3 points 4 months ago* (last edited 4 months ago) (1 children)

You'll have to wait for the connect dev for that unfortunately

[–] remotelove@lemmy.ca 3 points 4 months ago

It was kind of a stretch for it being an instance issue anyway, but thanks for the response. (These issues started after the last instance update, which does point the finger at the client in many cases. Not all, but many.)

Unfortunately, my manipulation skills are top tier. Unless you happen to have the memory span of a carrot, this issue report is now planted in your head as something to associate with any possible future issues that you might encounter. (I know how engineers think, bud.)

Ok, maybe I just have a little too much time on my hands at the moment to type out extended comments. ;)

[–] nixx@lemmy.ca 2 points 4 months ago (1 children)

I would love to see a write up on what was done. PostgeSQL is my favourite system and I would like to know more.

[–] Shadow@lemmy.ca 1 points 4 months ago

Nothing too crazy. My discovery this week is that our custom postgres conf actually wasn't being applied and I had never noticed. It got fixed a while ago but I haven't been following the upstream ansible repo - https://github.com/LemmyNet/lemmy-ansible/commit/22aa431a3d1ed8437a418633e9d92ae4af45bbf5

So just fixing up postgres to actually use the conf with our tuned wal & worker settings, along with the usual apt-get update/upgrade/reboot.

[–] MattW03@lemmy.ca 2 points 4 months ago